Packet processing for logical datapath sets

ABSTRACT

Some embodiments provide a method that processes network data through a network. The method receives a packet destined for a network host associated with a logical datapath set implemented by a set of managed edge switching elements and a set of managed non-edge switching elements in the network. The method determines whether the packet is a known packet. When the packet is a known packet, the method forwards the packet to a managed switching element in the set of managed edge switching elements for forwarding to the network host. When the packet is not a known packet, the method forwards the packet to a managed switching element in the set of managed non-edge switching elements for further processing.

CLAIM OF BENEFIT TO PRIOR APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.13/543,784, filed on Jul. 6, 2012, now issued as U.S. Pat. No.10,021,019. U.S. patent application Ser. No. 13/543,784 is acontinuation of U.S. patent application Ser. No. 13/290,054, filed onNov. 4, 2011, now issued as U.S. Pat. No. 9,680,750. U.S. patentapplication Ser. No. 13/290,054 is a continuation in part application ofU.S. patent application Ser. No. 13/177,535, filed on Jul. 6, 2011, nowissued as U.S. Pat. No. 8,750,164; a continuation in part of applicationof U.S. patent application Ser. No. 13/177,536, filed on Jul. 6, 2011,now issued as U.S. Pat. No. 8,959,215; and a continuation in partapplication of U.S. patent application Ser. No. 13/177,538, filed Jul.6, 2011, now issued as U.S. Pat. No. 8,830,823. U.S. patent applicationSer. Nos. 13/177,535, 13/177,536, and 13/177,538 claim benefit to U.S.Provisional Patent Application 61/361,912, filed on Jul. 6, 2010; U.S.Provisional Patent Application 61/361,913, filed on Jul. 6, 2010; U.S.Provisional Patent Application 61/429,753, filed on Jan. 4, 2011; U.S.Provisional Patent Application 61/429,754, filed on Jan. 4, 2011; U.S.Provisional Patent Application 61/466,453, filed on Mar. 22, 2011; U.S.Provisional Patent Application 61/482,205, filed on May 3, 2011; U.S.Provisional Patent Application 61/482,615, filed on May 4, 2011; U.S.Provisional Patent Application 61/482,616, filed on May 4, 2011; U.S.Provisional Patent Application 61/501,743, filed on Jun. 27, 2011; andU.S. Provisional Patent Application 61/501,785, filed on Jun. 28, 2011.U.S. patent application Ser. No. 13/543,784 claims benefit to U.S.Provisional Patent Application 61/482,205, filed on May 3, 2011, U.S.Provisional Patent Application 61/482,615, filed on May 4, 2011, U.S.Provisional Patent Application 61/482,616, filed on May 4, 2011, U.S.Provisional Patent Application 61/501,743, filed on Jun. 27, 2011, U.S.Provisional Patent Application 61/501,785, filed on Jun. 28, 2011, U.S.Provisional Patent Application 61/505,100, filed on Jul. 6, 2011, U.S.Provisional Patent Application 61/505,102, filed on Jul. 6, 2011, andU.S. Provisional Patent Application 61/505,103, filed on Jul. 6, 2011.U.S. patent application Ser. Nos. 13/543,784 , now issued as U.S. Pat.No. 10,021,019; and 13/177,535, now issued as U.S. Pat. No. 8,750,164;and U.S. Provisional Patent Applications 61/361,912, 61/361,913,61/429,753, 61/429,754, 61/466,453, 61/482,205, 61/482,615, 61/482,616,61/501,743, 61/501,785, 61/505,100, 61/505,102, and 61/505,103 areincorporated herein by reference.

BACKGROUND

Many current enterprises have large and sophisticated networkscomprising switches, hubs, routers, servers, workstations and othernetworked devices, which support a variety of connections, applicationsand systems. The increased sophistication of computer networking,including virtual machine migration, dynamic workloads, multi-tenancy,and customer specific quality of service and security configurationsrequire a better paradigm for network control. Networks havetraditionally been managed through low-level configuration of individualcomponents. Network configurations often depend on the underlyingnetwork: for example, blocking a user's access with an access controllist (“ACL”) entry requires knowing the user's current IP address. Morecomplicated tasks require more extensive network knowledge: forcingguest users' port 80 traffic to traverse an HTTP proxy requires knowingthe current network topology and the location of each guest. Thisprocess is of increased difficulty where the network switching elementsare shared across multiple users.

In response, there is a growing movement, driven by both industry andacademia, towards a new network control paradigm called Software-DefinedNetworking (SDN). In the SDN paradigm, a network controller, running onone or more servers in a network, controls, maintains, and implementscontrol logic that governs the forwarding behavior of shared networkswitching elements on a per user basis. Making network managementdecisions often requires knowledge of the network state. To facilitatemanagement decision-making, the network controller creates and maintainsa view of the network state and provides an application programminginterface upon which management applications may access a view of thenetwork state.

Three of the many challenges of large networks (including datacentersand the enterprise) are scalability, mobility, and multi-tenancy andoften the approaches taken to address one hamper the other. Forinstance, one can easily provide network mobility for virtual machines(VMs) within an L2 domain, but L2 domains cannot scale to large sizes.Also, retaining tenant isolation greatly complicates mobility. Despitethe high-level interest in SDN, no existing products have been able tosatisfy all of these requirements.

BRIEF SUMMARY

Some embodiments of the invention provide a method of distributingpacket processing across several managed non-edge switching elements. Insome embodiments, a managed edge switching element of several managedswitching elements that implement a logical datapath set may not be ableto process a packet through the logical datapath set so as to determinewhere to forward the packet. In such cases, the managed edge switchingelement forwards the packet to one of several managed non-edge switchingelements for further processing. The managed edge switching element maynot be able to process the packet for different reasons. For example,the packet may be unknown to the managed edge switching element (e.g.,the forwarding table of the managed switching element does not have anentry that matches the packet). As another example, the packet may be acertain type of packet (e.g., a multicast packet and broadcast packet).

To distribute the processing of packets across the managed non-edgeswitching elements, some embodiments determine a managed non-edgeswitching element based on a hash function. In some embodiments, themethod applies the hash function on the packet's header (e.g., sourcemedia access control (MAC) address, destination MAC address, etc.) togenerate a hash value. The method of some embodiments then compares thehash value to a hash range list that includes different hash valueranges that correspond to different managed non-edge switching elements.Based on the comparison, the method determines a managed non-edgeswitching element to which the packet is forwarded for furtherprocessing.

The preceding Summary is intended to serve as a brief introduction tosome embodiments of the invention. It is not meant to be an introductionor overview of all inventive subject matter disclosed in this document.The Detailed Description that follows and the Drawings that are referredto in the Detailed Description will further describe the embodimentsdescribed in the Summary as well as other embodiments. Accordingly, tounderstand all the embodiments described by this document, a full reviewof the Summary, Detailed Description and the Drawings is needed.Moreover, the claimed subject matters are not to be limited by theillustrative details in the Summary, Detailed Description and theDrawings, but rather are to be defined by the appended claims, becausethe claimed subject matters can be embodied in other specific formswithout departing from the spirit of the subject matters.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features of the invention are set forth in the appendedclaims. However, for purposes of explanation, several embodiments of theinvention are set forth in the following figures.

FIG. 1 conceptually illustrates a network architecture of someembodiments.

FIG. 2 conceptually illustrates a network control system of someembodiments that manages physical switching elements.

FIG. 3 conceptually illustrates a network control system of someembodiments for managing software switching elements.

FIG. 4 conceptually illustrates a network control system of someembodiments for managing physical and software switching elements.

FIG. 5 conceptually illustrates a network control system of someembodiments for managing edge switching elements and non-edge switchingelements.

FIG. 6 conceptually illustrates an example of a tunnel provided by atunneling protocol.

FIG. 7 illustrates the transmission of network data through a tunnelaccording to some embodiments of the invention.

FIG. 8 illustrates an example of multiple logical switching elementsimplemented across a set of switching elements.

FIG. 9 conceptually illustrates a block diagram of a switching elementof some embodiments.

FIG. 10 conceptually illustrates an architectural diagram of a hardwareswitching element of some embodiments.

FIG. 11 conceptually illustrates an architectural diagram of a computingdevice that includes a software switching element of some embodiments.

FIG. 12 conceptually illustrates an architectural diagram of a softwareswitching element of some embodiments.

FIG. 13 conceptually illustrates a network control system of someembodiments for managing a switching element.

FIG. 14 conceptually illustrates a processing pipeline of someembodiments for processing network data through a logical switchingelement.

FIG. 15 conceptually illustrates a process of some embodiments forprocessing network data.

FIG. 16 conceptually illustrates a network architecture of someembodiments that includes a pool node.

FIG. 17 conceptually illustrates an example multi-recipient packet flowthrough the network architecture illustrated in FIG. 16 according tosome embodiments of the invention

FIG. 18 conceptually illustrates another example multi-recipient packetflow through the network architecture illustrated in FIG. 16 accordingto some embodiments of the invention FIG. 19 conceptually illustrates anexample of a pool node configured to assist in processing packets formanaged switching elements.

FIG. 20 conceptually illustrates a process of some embodiments forprocessing packets.

FIG. 21 conceptually illustrates a network architecture of someembodiments that includes root nodes.

FIG. 22 conceptually illustrates an architectural diagram of a pool nodeof some embodiments.

FIG. 23 conceptually illustrates a network architecture of someembodiments that includes extenders.

FIG. 24 conceptually illustrates a network architecture that includes amanaged network zone and an unmanaged network zone.

FIG. 25 conceptually illustrates a network architecture that includes amanaged network zone and an unmanaged network zone, which are part of adata center.

FIG. 26 conceptually illustrates an example of mapping logical contexttags between managed networks and unmanaged networks.

FIG. 27 conceptually illustrates an architectural diagram of an extenderof some embodiments.

FIG. 28 conceptually illustrates a network architecture for distributingpacket processing between pool nodes.

FIG. 29 conceptually illustrates an example tunnel configuration of someembodiments.

FIG. 30 conceptually illustrates a process of some embodiments forprocessing packets.

FIG. 31 conceptually illustrates a block diagram of a switching elementof some embodiments that processes packets to determine a pool node towhich to send the packet.

FIG. 32 conceptually illustrates a process of some embodiments forcreating a managed network.

FIG. 33 conceptually illustrates the creation of additional switchingelements to a managed network according to some embodiments of theinvention.

FIG. 34 conceptually illustrates the addition of managed switchingelements and the creation of additional switching elements to a managednetwork according to some embodiments of the invention.

FIG. 35 conceptually illustrates an example of updating hash functionswhen a pool node is added to a managed network.

FIG. 36 conceptually illustrates a process of some embodiments forupdating a hash function.

FIGS. 37A-F conceptually illustrate examples of pool node failurehandling according to some embodiments of the invention.

FIGS. 38A-B conceptually illustrate the creation of additional networkcontrollers to manage a managed network according to some embodiments ofthe invention.

FIG. 39 conceptually illustrates a process of some embodiments forprocessing a packet through a logical switching element that isimplemented across a set of managed switching elements in a managednetwork.

FIG. 40 conceptually illustrates a processing pipeline of someembodiments for processing a packet through a logical switching element.

FIG. 41 conceptually illustrates a processing pipeline of someembodiments for processing a packet through a logical switching element.

FIG. 42 conceptually illustrates distribution of logical processingacross managed switching elements in a managed network according to someembodiments of the invention.

FIG. 43 conceptually illustrates distribution of logical processingacross managed switching elements in a managed network according to someembodiments of the invention.

FIG. 44 illustrates several example flow entries that implement aportion of a processing pipeline of some embodiments.

FIG. 45 conceptually illustrates a network architecture of someembodiments.

FIG. 46 conceptually illustrates an electronic computer system withwhich some embodiments of the invention are implemented.

FIGS. 47A-C conceptually illustrate an example of network controllerfailure handling according to some embodiments of the invention.

FIGS. 48A-C conceptually illustrate another example of networkcontroller failure handling according to some embodiments of theinvention.

DETAILED DESCRIPTION

In the following detailed description of the invention, numerousdetails, examples, and embodiments of the invention are set forth anddescribed. However, it will be clear and apparent to one skilled in theart that the invention is not limited to the embodiments set forth andthat the invention may be practiced without some of the specific detailsand examples discussed.

I. Environment

The following section will describe the environment in which someembodiments of the inventions are implements. In the presentapplication, switching elements and machines may be referred to asnetwork elements. In addition, a network that is managed by one or morenetwork controllers may be referred to as a managed network in thepresent application. In some embodiments, the managed network includesonly managed switching elements (e.g., switching elements that arecontrolled by one or more network controllers) while, in otherembodiments, the managed network includes managed switching elements aswell as unmanaged switching elements (e.g., switching elements that arenot controlled by a network controller).

FIG. 1 conceptually illustrates a network architecture 100 of someembodiments. As shown, the network architecture 100 includes networkcontrollers 110 and 120, managed switching elements 130-150, andmachines 155-185.

In some embodiments, the managed switching elements 130-150 routenetwork data (e.g., packets) between network elements in the networkthat are coupled to the managed switching elements 130-150. Forinstance, the managed switching element 130 routes network data betweenthe machines 155-165 and the managed switching element 140. Similarly,the managed switching element 140 routes network data between themachine 170 and the managed switching elements 140 and 150, and themanaged switching element 150 routes network data between the machines175-185 and the managed switching element 150.

The managed switching elements 130-150 of some embodiments can beconfigured to route network data according to defined rules. In someembodiments, the managed switching elements 130-150 routes network databased on routing criteria defined in the rules. Examples of routingcriteria include source media access control (MAC) address, destinationMAC, packet type, source Internet Protocol (IP) address, destination IPaddress, source port, destination port, and/or virtual local areanetwork (VLAN) identifier, among other routing criteria.

In some embodiments, the managed switching elements 130-150 can includestandalone physical switching elements, software switching elements thatoperate within a computer, or any other type of switching element. Forexample, each of the managed switching elements 130-150 may beimplemented as a hardware switching element, a software switchingelement, a virtual switching element, a network interface controller(NIC), or any other type of network element that can route network data.Moreover, the software or virtual switching elements may operate on adedicated computer, or on a computer that performs non-switchingoperations.

The machines 155-185 send and receive network data between each otherover the network. In some embodiments, the machines 155-185 are referredto as network hosts that are each assigned a network layer hostaddresses (e.g., IP address). Some embodiments refer to the machines155-185 as end systems because the machines 155-185 are located at theedge of the network. In some embodiments, each of the machines 155-185can be a desktop computer, a laptop computer, a smartphone, a virtualmachine (VM) running on a computing device, a terminal, or any othertype of network host.

In some embodiments, each of the network controllers 110 and 120controls one or more managed switching elements 130-150 that are locatedat the edge of a network (e.g., edge switching elements or edgedevices). In this example, the managed switching elements 130-150 areedge switching elements. That is, the managed switching elements 130-150are switching elements that are located at or near the edge of thenetwork. In some embodiments, an edge switching element is the lastswitching element before end machines (the machines 155-185 in thisexample) in a network. As indicated by dashed arrows in FIG. 1 , thenetwork controller 110 controls (i.e., manages) switching elements 130and 140 and the network controller 120 controls switching element 150.In this application, a switching element that is controlled by a networkcontroller of some embodiments may be referred to as a managed switchingelement.

Controlling only edge switches allows the network architecture 100 to bedeployed independent of concerns about the hardware vendor of thenon-edge switches, because deploying at the edge allows the edgeswitches to treat the internal nodes of the network as simply acollection of elements that moves packets without considering thehardware makeup of these internal nodes. Also, controlling only edgeswitches makes distributing switching logic computationally easier.Controlling only edge switches also enables non-disruptive deployment ofthe system because edge-switching solutions can be added as top of rackswitches without disrupting the configuration of the non-edge switches.

In addition to controlling edge switching elements, the networkcontrollers 110 and 120 of some embodiments also utilize and controlnon-edge switching elements (e.g., pool nodes, root nodes, andextenders, which are described in further detail below) that areinserted in the network to simplify and/or facilitate the operation ofthe managed edge switching elements. For instance, in some embodiments,the network controller 110 and 120 require the switching elements thatthe network controller 110 and 120 control to be interconnected in ahierarchical switching architecture that has several edge switchingelements as the leaf nodes in the hierarchical switching architectureand one or more non-edge switching elements as the non-leaf nodes inthis architecture. In some such embodiments, each edge switching elementconnects to one or more of the non-leaf switching elements, and usessuch non-leaf switching elements to facilitate the communication of theedge switching element with other edge switching elements. Examples ofsuch communications with an edge switching elements in some embodimentsinclude (1) routing of a packet with an unknown destination address(e.g., unknown MAC address) to the non-leaf switching element so thatthe non-leaf switching element can route the packet to the appropriateedge switching element, (2) routing a multicast or broadcast packet tothe non-leaf switching element so that the non-leaf switching elementcan distribute the multicast or broadcast packet to the desireddestinations.

Some embodiments employ one level of non-leaf (non-edge) switchingelements that connect to edge switching elements and in some cases toother non-leaf switching elements. Other embodiments, on the other hand,employ multiple levels of non-leaf switching elements, with each levelof non-leaf switching elements after the first level serving as amechanism to facilitate communication between lower level non-leafswitching elements and leaf switching elements. In some embodiments, thenon-leaf switching elements are software switching elements that areimplemented by storing the switching tables in the memory of astandalone computer instead of an off the shelf switch. In someembodiments, the standalone computer may also be executing in some casesa hypervisor and one or more virtual machines on top of that hypervisor.Irrespective of the manner by which the leaf and non-leaf switchingelements are implemented, the network controllers 110 and 120 of someembodiments store switching state information regarding the leaf andnon-leaf switching elements.

As mentioned above, the switching elements 130-150 of some embodimentsroute network data between network elements in the network. In someembodiments, the network controllers 110 and 120 configure the managedswitching elements 130-150 s′ routing of network data between thenetwork elements in the network. In this manner, the network controllers110 and 120 can control the flow (i.e., specify the datapath) of networkdata between network elements.

For example, the network controller 110 might instruct the managedswitching elements 130 and 140 to route network data from the machine155 to the machine 170 (and vice versa) and to not route (e.g., drop)network data from other machines to the machines 155 and 170. In suchcase, the network controller 110 controls the flow of network datathrough the managed switching elements 130 and 140 such that networkdata transmitted to and from the machine 155 is only routed to themachine 170. Thus, the machines 155 and 170 cannot send and receivenetwork data to and from the machines 160, 165, and 175-185.

In some embodiments, the network controllers 110 and 120 store physicalnetwork information and logical network information. The physicalnetwork information specifies the physical components in the managednetwork and how the physical components are physically connected oneanother in the managed network. For example, the physical networkinformation may include the number of machines, managed switchingelements, pool nodes, root nodes, and extenders (the latter three aredescribed in further detail in the following sections), and how thecomponents are physically connected to one another in the managednetwork. The logical network information may specify the logicalconnections between a set of physical components in the managed network(e.g., machines) and a mapping of the logical connections across thephysical components of the managed network.

Some embodiments of the network controllers 110 and 120 implement alogical switching element across the managed switching elements 130-150based on the physical network information and the logical switchingelement information described above. A logical switching element can bedefined to function any number of different ways that a switchingelement might function. The network controllers 110 and 120 implementthe defined logical switching element through control of the managedswitching elements 130-150. In some embodiments, the network controllers110 and 120 implement multiple logical switching elements across themanaged switching elements 130-150. This allows multiple differentlogical switching elements to be implemented across the managedswitching elements 130-150 without regard to the network topology of thenetwork.

In some embodiments, a logical datapath set defines a logical switchingelement. A logical datapath set, in some embodiments, is a set ofnetwork datapaths through the managed switching elements 130-150 thatimplement the logical switching element and the logical switch's definedfunctionalities. In these embodiments, the network controllers 110 and120 translate (e.g., maps) the defined logical datapath set into networkconfiguration information for implementing the logical switchingelement. The network controllers 110 and 120 translate the definedlogical datapath set into a corresponding set of data flows (i.e.,datapaths) between network elements in the network, in some embodiments.In these instances, the network controllers 110 and 120 instruct themanaged switching elements 130-150 to route network data according tothe data flows and, thus, implement the functionalities of the definedlogical switching element.

Different embodiments of the network controllers 110 and 120 areimplemented differently. For example, some embodiments implement thenetwork controllers 110 and 120 in software as instances of a softwareapplication. In these cases, the network controllers 110 and 120 may beexecuted on different types of computing devices, such as a desktopcomputer, a laptop computer, a smartphone, etc. In addition, thesoftware application may be executed on a virtual machine that runs on acomputing device in some embodiments. In some embodiments, the networkcontrollers 110 and 120 are implemented in hardware (e.g., circuits).

As mentioned above by reference to FIG. 1 , the managed switchingelements controlled by network controllers of some embodiments may bephysical switching elements. FIG. 2 illustrates an example of a networkcontrol system that includes physical switching elements. This figureconceptually illustrates a network control system 200 of someembodiments for managing physical switching elements. Specifically, thenetwork control system 200 manages network data in a data center thatincludes top of the rack (TOR) switching elements 230-250 and racks ofhosts 260-280. Network controllers 210 and 220 manage the network bycontrolling the TOR switching elements 230-250.

A TOR switching element, in some embodiments, routes network databetween hosts in the TOR switch's rack and network elements coupled tothe TOR switching element. In the example illustrated in FIG. 2 , theTOR switching element 230 routes network data between the rack of hosts260 and TOR switching elements 240 and 250, the TOR switching element240 routes network data between the rack of hosts 270 and TOR switchingelements 230 and 250, and the TOR switching element 250 routes networkdata between the rack of hosts 280 and TOR switching elements 230 and240.

As shown, each rack of hosts 260-280 includes multiple hosts. The hostsof some embodiments in the racks of hosts 260-280 are physical computingdevices. In some embodiments, each host is a computing device that isassigned a network layer host address (e.g., IP address). The hosts ofsome embodiments send and receive network data to and from each otherover the network.

As mentioned above, the network controller of some embodiments can beimplemented in software as an instance of an application. As illustratedin FIG. 2 , the network controllers 210 and 220 are instances of asoftware application. As shown, each of the network controllers 210 and220 includes several software layers: a control application layer, avirtualization application layer, and a networking operating systemlayer.

In some embodiments, the control application layer receives user inputthat specifies a network switching element. The control applicationlayer may receive the user input in any number of different interfaces,such as a graphical user interface (GUI), a command line interfaces, aweb-based interface, a touchscreen interface, etc. In some embodiments,the user input specifies characteristics and behaviors of the networkswitching element, such as the number of switching element ports, accesscontrol lists (ACLs), network data forwarding, port security, or anyother network switching element configuration options.

The control application layer of some embodiments defines a logicaldatapath set based on user input that specifies a network switchingelement. As noted above, a logical datapath set is a set of networkdatapaths through managed switching elements that are used to implementthe user-specified network switching element. In other words, thelogical datapath set is a logical representation of the networkswitching element and the network switch's specified characteristics andbehaviors.

Some embodiments of the virtualization application layer translate thedefined logical datapath set into network configuration information forimplementing the logical network switching element across the managedswitching elements in the network. For example, the virtualizationapplication layer of some embodiments translates the defined logicaldatapath set into a corresponding set of data flows. In some of thesecases, the virtualization application layer may take into accountvarious factors (e.g., logical switching elements that are currentlyimplemented across the managed switching elements, the current networktopology of the network, etc.), in determining the corresponding set ofdata flows.

The network operating system layer of some embodiments configures themanaged switching elements' routing of network data. In someembodiments, the network operating system instructs the managedswitching elements to route network data according to the set of dataflows determined by the virtualization application layer.

In some embodiments, the network operating system layer maintainsseveral views of the network based on the current network topology. Oneview that the network operating system layer maintains is a logicalview. The logical view of the network includes the different logicalswitching elements that are implemented across the managed switchingelements, in some embodiments. Some embodiments of the network operatingsystem layer maintain a managed view of the network. Such managed viewsinclude the different managed switching elements in the network (i.e.,the switching elements in the network that the network controllerscontrol). In some embodiments, the network operating system layer alsomaintains relationship data that relate the logical switching elementsimplemented across the managed switching elements to the managedswitching elements.

While FIG. 2 (and other figures in this application) may show a set ofmanaged switching elements managed by a network controller, someembodiments provide several network controllers (also referred to as acluster of network controllers or a control cluster) for managing theset of managed switching elements. In other embodiments, differentcontrol clusters may manage different sets of managed switchingelements. Employing a cluster of network controllers in such embodimentsto manage a set of managed switches increases the scalability of themanaged network and increases the redundancy and reliability of themanaged network. In some embodiments, the network controllers in acontrol cluster share (e.g., through the network operating system layerof the network controllers) data related to the state of the managednetwork in order to synchronize the network controllers.

FIG. 3 conceptually illustrates a network control system 300 of someembodiments for managing software switching elements. As shown, thenetwork control system 300 includes network controllers 310 and 320, TORswitching elements 330-350, and racks of hosts 360-380.

The TOR switching elements 330-350 are similar to the TOR switchingelements 230-250. The TOR switching elements 330-350 route network databetween network elements in the network that are coupled to the TORswitching elements 330-350. In this example, the TOR switching element330 routes network data between the rack of hosts 360 and TOR switchingelements 340 and 350, the TOR switching element 340 routes network databetween the rack of hosts 370 and TOR switching elements 330 and 350,and the TOR switching element 350 routes network data between the rackof hosts 380 and TOR switching elements 330 and 340. Since the TORswitching elements 330-350 are not managed switching elements, thenetwork controllers 310 and 320 do not control these switching elements.Thus, the TOR switching elements 330-350 rely on the switching elements'preconfigured functionalities to route network data.

As illustrated in FIG. 3 , each host in the racks of hosts 360-380includes a software switching element (an open virtual switch (OVS) inthis example) and several VMs. The VMs are virtual machines that areeach assigned a set of network layer host addresses (e.g., a MAC addressfor network layer 2, an IP address for network layer 3, etc.) and cansend and receive network data to and from other network elements overthe network.

The OVSs of some embodiments route network traffic between networkelements coupled to the OVSs. For example, in this example, each OVSroutes network data between VMs that are running on the host on whichthe OVS is running, OVSs running on other hosts in the rack of hosts,and the TOR switching element of the rack.

By running a software switching element and several VMs on a host, thenumber of end machines or network hosts in the network may increase.Moreover, when a software switching element and several VMs are run onhosts in the racks of hosts 360-380, the network topology of the networkis changed. In particular, the TOR switching elements 330-350 are nolonger edge switching elements. Instead, the edge switching elements inthis example are the software switching elements running on the hostssince these software switching elements are the last switching elementsbefore end machines (i.e., VMs in this example) in the network.

The network controllers 310 and 320 perform similar functions as thenetwork controllers 210 and 220, which described above by reference toFIG. 2 , and also are for managing edge switching elements. As such, thenetwork controllers 310 and 320 manage the OVSs that are running on thehosts in the rack of hosts 360-380.

The above FIGS. 2 and 3 illustrate a network control systems formanaging physical switching elements and a network control system formanaging software switching elements, respectively. However, the networkcontrol system of some embodiments can manage both physical switchingelements and software switching elements. FIG. 4 illustrates an exampleof such a network control system. In particular, this figureconceptually illustrates a network control system 400 of someembodiments for managing TOR switching element 430 and OVSs running onhosts in the racks of hosts 470 and 480.

The network controllers 410 and 420 perform similar functions as thenetwork controllers 210 and 220, which described above by reference toFIG. 2 , and also are for managing edge switching elements. In thisexample, the managed switching element 430 and the OVSs running on thehosts in the racks of hosts 470 and 480 are edge switching elementsbecause they are the last switching elements before end machines in thenetwork. In particular, the network controller 410 manages the TORswitching element 410 and the OVSs that are running on the hosts in therack of hosts 460, and the network controller 420 manage the OVSs thatare running on the hosts in the rack of hosts 480.

The above figures illustrate examples of network controllers thatcontrol edge switching elements in a network. However, in someembodiments, the network controllers can control non-edge switchingelements as well. FIG. 5 illustrates a network control system thatincludes such network controllers. In particular, FIG. 5 conceptuallyillustrates a network control system 500 of some embodiments formanaging TOR switching elements 530-550 and OVS running on hosts in theracks of hosts 570 and 580.

As shown in FIG. 5 , the network controllers 510 and 520 manage edgeswitching elements and non-edge switching elements. Specifically, thenetwork controller 510 manages the TOR switching elements 530 and 520,and the OVSs running on the hosts in the rack of hosts 570. The networkcontroller 520 manages TOR switching element 580 and the OVSs running onthe hosts in the rack of hosts 580. In this example, the TOR switchingelement 530 and the OVSs running on the hosts in the racks of hosts 570and 580 are edge switching elements, and the TOR switching elements 540and 550 are non-edge switching elements. The network controllers 510 and520 perform similar functions as the network controllers 210 and 220,which are described above by reference to FIG. 2 .

II. Network Constructs

The following section describes several network constructs. Differentembodiments described in this application may utilize one or more ofthese network constructs to facilitate some or all of thefunctionalities of the different embodiments.

FIG. 6 conceptually illustrates an example of a tunnel provided by atunneling protocol. As shown in FIG. 6 , a network 600 includes routers610 and 620, switching elements 630 and 640, and machines 650-680. Themachines 650-680 are similar to the machines 155-185 described above.

The machines 650-680 of some embodiments are network hosts that are eachassigned a set of network layer host addresses (e.g., a MAC address fornetwork layer 2, an IP address for network layer 3, etc.). The machines650-680 may also be referred to as end machines. Similar to the machines155-185 described above, each of the machines 650-680 can be a desktopcomputer, a laptop computer, a smartphone, a virtual machine (VM)running on a computing device, a terminal, or any other type of networkhost. In addition, the machines 650-680 may belong to different tenants(e.g., in a data center environment). As illustrated in FIG. 6 , each ofthe machines 650-680 belongs to either tenant A or tenant B.

The switching elements 630 and 640 are network switching elements thatroute (e.g., forwards) network data at the data link layer (alsoreferred to as layer 2 or L2 layer) based on protocols such as theEthernet protocol. The switching elements 630 and 640 may also bereferred to as network bridges in some embodiments. As shown, theswitching element 630 routes network data at the data link layer betweenthe machines 650 and 660 and the router 610, and the switching element640 routes network data at the data link layer between the machines 670and 680 and the router 620.

To route network data at the data link layer, some embodiments of theswitching elements 630 and 640 use a media access control (MAC) addressof a network host's network interface card (NIC) to determine where toroute network data (e.g., packets, frames, etc.). The switching elements630 and 640 are implemented differently in different embodiments. Forinstance, each of the switching elements 630 and 640 can be implementedas a hardware switching element, a software switching element, a virtualswitching element, some types of network interface card (NIC), or anyother type of network element that can route network data at the datalink layer.

Furthermore, the switching elements 630 and 640 support any number ofdifferent types of tunneling protocols in different embodiments. Asshown, examples of tunneling protocols include control and provisioningof wireless access points (CAPWAP), generic route encapsulation (GRE),GRE Internet Protocol Security (IPsec), among other types of tunnelingprotocols.

The routers 610 and 620 are network routers that route network data atthe network layer (also referred to as the layer 3 or L3 layer) based onprotocols such as the Internet Protocol (IP). As illustrated in FIG. 6 ,the router 610 routes network data at the network layer between therouter 620 and the switching element 630, and the router 620 routesnetwork data at the network layer between the router 610 and theswitching element 640.

In order to route network data at the network layer, the routers 610 and620 of some embodiments use an IP address assigned to a network host todetermine where to route network data (e.g., packets). Moreover, therouters 610 and 620 of some embodiments may provide other functions aswell, such as security functions, quality of service (QoS) functions,checksum functions, flow accounting functions, or any other type ofrouter functions.

Different embodiments of the routers 610 and 620 can be implementeddifferently. For example, each of the routers 610 and 620 can beimplemented as a hardware router, a software router, a virtual router,or any other type of network element that can route network data at thenetwork layer.

As mentioned above, the switching elements 630 and 640 of someembodiments can support tunneling protocols. In some embodiments, atunneling protocol allows network data to be sent along a path betweentwo points in a network where the tunneling protocol used by the networkelements along the path in the network is different than the payloadprotocol used by the destination network element

In some embodiments, a tunneling protocol is a network protocol (e.g., adelivery protocol) that encapsulates another protocol (e.g., a payloadprotocol). A tunneling protocol can be used, for example, to transmitnetwork data over an incompatible delivery-network. For instance, inthis example, a tunneling protocol may provide a tunnel over a layer 3network through which layer 2 network data is transmitted. As such, fromthe perspective of the machines 650-680, the machines 650-680 arecommunicating over an L2 network. In other words, a tunneling protocolfacilitates the communication of layer 2 network data between networkhosts separated by a layer 3 network.

FIG. 6 illustrates a tunnel 690 that has been established between theswitching element 630 and the switching element 640. As shown, thetunnel 690 is established over a layer 3 network 695 (e.g., theInternet). The tunnel 690 allows layer 2 network data to be transmittedbetween the machines 650-680 by encapsulating the layer 2 network datawith a layer 3 header and transmitting the network data through thetunnel 690 that is established over the layer 3 network 695.

When the switching elements 630 and 640 route packets between eachother, the packets are routed through the tunnel 690 over the network695. In some instances, the network 695 includes switching elements (notshown in FIG. 6 ) that facilitate the forwarding of packets through thetunnel. Examples of such switching elements in the network 695 includestandard switches (which may be cheaper than proprietary switches), offthe shelf switches, or any other type of switching element that forwardsdata through the network 695. These switching elements may be referredto as unmanaged switching elements because they are not managed by anetwork controller that manages the switching elements 630 and 640.

Using the tunnel 690 allows the switching elements 630 and 640 to routepackets through the network 695 of switching elements independent of thetype of switching elements in the network 695. The switching elements630 and 640 treat the switching elements in the network 695 as simply acollection of elements that moves packets without considering thehardware makeup of these switching elements. Thus, the use of the tunnel690 to route packets between the switching elements 630 and 640 does notdisrupt the network 695. For instance, the switching elements in thenetwork 695 do not have to be managed in order to facilitatecommunication between the switching elements 630 and 640.

As noted above, packets that are routed through the tunnel 690 areencapsulated with a tunneling protocol. As such, the switching elementsincluded in the network 695 are unaware of the encapsulated data (e.g.,logical network data such as a MAC address or an IP address) that isrouted through the tunnel 690. In other words, the encapsulated data ishidden from the unmanaged switching elements in the network 695 thatforward the encapsulated data through the network 695.

In some embodiments, routing packets through the tunnel 690 requiresfewer entries in the forwarding tables of the unmanaged switchingelements in the network 695 compared to routing the packets usingnon-tunneling methods (e.g., using a transmission control protocol(TCP), a user datagram protocol (UDP), or an Ethernet protocol). This isbecause, in the former case, an unmanaged switching element learns andstores a pair of entries (e.g., an entry for packets that are routed inone direction of the tunnel 690 and an entry for packets that are routedin the other direction of the tunnel 690) for packets that are routedthrough the tunnel 690. In the latter case, an unmanaged switchingelement learns and stores forwarding entries for each of the differentpackets that are forwarded through the unmanaged switching elements.Accordingly, routing packets through the tunnel 690 reduces the size ofthe forwarding tables of the unmanaged switching elements in the network695.

As shown in FIG. 6 , a single tunnel 690 is established between theswitching elements 630 and 640. However, in some embodiments multipletunnels using the same or different tunneling protocols may beestablished between the switching elements 630 and 640. For example, thetunnel 690 shown in FIG. 6 is a bidirectional tunnel, as indicated by anarrow at each end of the tunnel 690. However, some embodiments mayprovide unidirectional tunnels. In such cases, a tunnel is establishedfor each direction of communication between two points in the network.Referring to FIG. 6 as an example, when one of the machines 650 and 660wishes to communicate with one of the machines 670 and 680, a tunnel isestablished that allows network data to be transmitted only from theswitching element 630 to the switching element 640. Conversely, when oneof the machines 670 and 680 wishes to communicate with one of themachines 650 and 660, a tunnel is established that allows network datato be transmitted from only the switching element 640 to the switchingelement 630.

Although FIG. 6 illustrates routers and switching elements as separatecomponents, the functions described above for the router and switchingelements may be performed by a single component in some embodiments. Forinstance, some embodiments combine the functions of the router 610 andthe switching element 630 into one component and/or combine thefunctions of the router 620 and the switching element 640 into anothercomponent.

FIG. 7 illustrates the transmission of network data through a tunnelaccording to some embodiments of the invention. Specifically, FIG. 7conceptually illustrates multiplexing network data that belongs todifferent tenants through a tunnel 770. As shown, this figureillustrates a network 700 that includes switching elements 710 and 720and machines 730-760. The machines 730-760 are similar to the machines155-185 described above.

As illustrated in FIG. 7 , the tunnel 770 is established between theswitching element 710 and the switching element 720. For this example,the tunnel 770 is a unidirectional tunnel, as indicated by an arrow,that allows network data to be transmitted from the switching element710 to the switching element 720. As described above, differenttunneling protocols (e.g., CAPWAP, GRE, etc.) can be used to establishthe tunnel 770 in different embodiments.

When transmitting network data through the tunnel 770, some embodimentsinclude an identifier (ID) tag with the network data when the networkdata is transmitted through the tunnel 770. In some embodiments, an IDtag is a unique identifier for identifying a tenant to which the networkdata is associated. In this manner, switching elements can identify thetenant to which the network data belongs. This enables network data fordifferent tenants to be transmitted through a single tunnel. In someembodiments, an ID tag allows machines of different tenants to haveoverlapping network identifiers (e.g., logical MAC addresses or logicalIP addresses). For example, in a layer 2 network where some machines ofdifferent tenants each has the same MAC address, an ID tag can be usedto differentiate between the machines of the different tenants and thenetwork data directed at the different tenants. Similarly, an ID tag maybe used to differentiate between machines of different tenants wheresome of the machines of the different tenants each has the same IPaddress.

The following will describe an example of transmitting network databelonging to different tenants that have overlapping network identifiersthrough a single tunnel by reference to FIG. 7 . In this example, an IDtag “ID 1” is associated with tenant A and an ID tag “ID 2” isassociated with tenant B. As such, the switching elements 710 and 720are configured with this ID tag information (e.g., stored in a lookuptable). In addition, tenant A's machines and tenant B's machines haveoverlapping network identifiers (e.g., they have the same MAC addressesor are use the same private IP address space).

When the machine 730 sends packet A to machine 750, the packet A istransmitted to the switching element 710. When the switching element 710receives the packet A, the switching element 710 determines that thepacket A originated from a machine that belongs to tenant A (e.g., basedon the packet A's source MAC address and/or the port through which thepacket A is received). Then, the switching element 710 identifies the IDtag (e.g., by performing a lookup on a lookup table) that is associatedwith tenant A (ID 1 in this example) and includes the ID tag in thepacket A before the packet is transmitted to the switching element 720through the tunnel 770. Since tenant A's machine (machine 750) andtenant B's machine (machine 760) have overlapping network identifiers(e.g., the machine 750 and 760 each has the same MAC address or use thesame private IP address space), the switching element 720 would not beable to differentiate between tenant A's machines and tenant B'smachines based only on the machines' network identifiers. However, theID tag allows the switching element 720 to differentiate between tenantA's machines and tenant B's machines. Therefore, when the switchingelement 720 receives the packet A from the switching element 710 throughthe tunnel 770, the switching element 720 examines the ID tag includedin the packet A and determines the tenant to which the packet A belongs(e.g., by performing a lookup on a lookup table). After determining thetenant to which the packet A belongs, the switching element 720 removesthe ID tag from the packet A and transmits to the packet A to themachine 750, the intended recipient of the packet A in this example.

When the machine 740 sends packet B to machine 760, the switchingelements 710 and 720 perform similar functions as those performed forthe packet A described above. That is, the switching element 710determines the tenant to which the packet B belongs, identifies the IDtag associated with the tenant, and includes the ID tag in the packet B.Then, the switching element 710 transmits the packet B to the switchingelement 720 through the tunnel 770. When the switching element 720receives the packet B from the switching element 710 through the tunnel770, the switching element 720 determines the tenant to which the packetB belongs by examining the ID tag included in the packet, removes the IDtag from the packet B, and transmits the packet B to the machine 760. Asexplained, the ID tag allows network data for tenants A's machines andtenant B's machines, which have overlapping network identifiers, to betransmitted through a single tunnel 770.

As mentioned above, the managed switching elements of some embodimentscan be configured to route network data based on different routingcriteria. In this manner, the flow of network data through switchingelements in a network can be controlled in order to implement multiplelogical switching elements across the switching elements.

FIG. 8 illustrates an example of multiple logical switching elementsimplemented across a set of switching elements. In particular, FIG. 8conceptually illustrates logical switching elements 870 and 880implemented across switching elements 810-830. As shown in FIG. 8 , anetwork 800 includes switching elements 810-830 and machines 840-865.The machines 840-865 are similar to the machines 155-185 describedabove. As indicated in this figure, the machines 840, 850, and 860belong to tenant A and the machines 845, 855, and 865 belong to tenantB.

The switching elements 810-830 of some embodiments route network data(e.g., packets, frames, etc.) between network elements in the networkthat are coupled to the switching elements 810-830. As shown, theswitching element 810 routes network data between the machines 840 and845 and the switching element 820. Similarly, the switching element 810routes network data between the machine 850 and the switching elements810 and 820, and the switching element 830 routes network data betweenthe machines 855-865 and the switching element 820.

Moreover, each of the switching elements 810-830 routes network databased on the switch's forwarding tables. In some embodiments, aforwarding table determines where to route network data (e.g., a port onthe switch) according to routing criteria. For instance, a forwardingtable of a layer 2 switching element may determine where to routenetwork data based on MAC addresses (e.g., source MAC address and/ordestination MAC address). As another example, a forwarding table of alayer 3 switching element may determine where to route network databased on IP addresses (e.g., source IP address and/or destination IPaddress). Many other types of routing criteria are possible.

As shown in FIG. 8 , the forwarding table in each of the switchingelements 810-830 includes several records. In some embodiments, each ofthe records specifies operations for routing network data based onrouting criteria. The records may be referred to as flow entries in someembodiments as the records control the “flow” of data through theswitching elements 810-830.

FIG. 8 also illustrates conceptual representations of each tenant'slogical network. As shown, the logical network 880 of tenant A includesa logical switching element 885 to which tenant A's machines 840, 850,and 860 are coupled. Tenant B's logical network 890 includes a logicalswitching element 895 to which tenant B's machines 845, 855, and 865 arecoupled. As such, from the perspective of tenant A, tenant A has aswitching element to which only tenant A's machines are coupled, and,from the perspective of tenant B, tenant B has a switching element towhich only tenant B's machines are coupled. In other words, to eachtenant, the tenant has its own network that includes only the tenant'smachines.

The following will describe the conceptual flow entries for implementingthe flow of network data originating from the machine 840 and destinedfor the machine 850 and originating from the machine 840 and destinedfor the machine 860. First, the flow entries for routing network dataoriginating from the machine 840 and destined for the machine 850 willbe described followed by the flow entries for routing network dataoriginating from the machine 840 and destined for the machine 860.

The flow entry “A1 to A2” in the switching element 810's forwardingtable instructs the switching element 810 to route network data thatoriginates from machine 810 and is destined for the machine 850 to theswitching element 820. The flow entry “A1 to A2” in the forwarding tableof the switching element 820 instructs the switching element 820 toroute network data that originates from machine 810 and is destined forthe machine 850 to the machine 850. Therefore, when the machine 840sends network data that is destined for the machine 850, the switchingelements 810 and 820 route the network data along datapath 870 based onthe corresponding records in the switching elements' forwarding tables.

Furthermore, the flow entry “A1 to A3” in the switching element 810'sforwarding table instructs the switching element 810 to route networkdata that originates from machine 810 and is destined for the machine850 to the switching element 820. The flow entry “A1 to A3” in theforwarding table of the switching element 820 instructs the switchingelement 820 to route network data that originates from machine 810 andis destined for the machine 860 to the switching element 830. The flowentry “A1 to A3” in the forwarding table of the switching element 830instructs the switching element 830 to route network data thatoriginates from machine 810 and is destined for the machine 860 to themachine 860. Thus, when the machine 840 sends network data that isdestined for the machine 860, the switching elements 810-830 route thenetwork data along datapath 875 based on the corresponding records inthe switching elements' forwarding tables.

While conceptual flow entries for routing network data originating fromthe machine 840 and destined for the machine 850 and originating fromthe machine 840 and destined for the machine 860 are described above,similar flow entries would be included in the forwarding tables of theswitching elements 810-830 for routing network data between othermachines in tenant A's logical network 880. Moreover, similar flowentries would be included in the forwarding tables of the switchingelements 810-830 for routing network data between the machines in tenantB's logical network 890.

In some embodiments, tunnels provided by tunneling protocols describedabove may be used to facilitate the implementation of the logicalswitching elements 885 and 895 across the switching elements 810-830.The tunnels may be viewed as the “logical wires” that connect machinesin the network in order to implement the logical switching elements 880and 890. In some embodiments, unidirectional tunnels are used. Forinstance, a unidirectional tunnel between the switching element 810 andthe switching element 820 may be established and through which networkdata originating from the machine 840 and destined for the machine 850is transmitted. Similarly, a unidirectional tunnel between the switchingelement 810 and the switching element 830 may be established and throughwhich network data originating from the machine 840 and destined for themachine 860 is transmitted. In some embodiments, a unidirectional tunnelis established for each direction of network data flow between twomachines in the network.

Alternatively, or in conjunction with unidirectional tunnels,bidirectional tunnels can be used in some embodiments. For instance, insome of these embodiments, only one bidirectional tunnel is establishedbetween two switching elements. Referring to FIG. 8 as an example, atunnel would be established between the switching elements 810 and 820,a tunnel would be established between the switching elements 820 and830, and a tunnel would be established between the switching elements810 and 830. In some embodiments, ID tags are utilized to distinguishbetween the network data of different tenants (e.g., tenants A and B inFIG. 8 ), as described above by reference to FIG. 7 .

Configuring the switching elements in the various ways described aboveto implement multiple logical switching elements across a set ofswitching elements allows multiple tenants, from the perspective of eachtenant, to each have a separate network and/or switching element whilethe tenants are in fact sharing some or all of the same set of switchingelements and/or connections between the set of switching elements (e.g.,tunnels, physical wires).

FIG. 9 conceptually illustrates a block diagram of a switching element900 of some embodiments. Many of the switching elements illustrated inthe figures throughout this application may be the same or similar tothe switching element 900 as described below. As illustrated in thisfigure, the switching element 900 includes ingress ports 910, egressports 920, dispatch port 930, and a forwarding table 940.

The ingress ports 910 conceptually represent a set of ports throughwhich the switching element 900 receives network data. The ingress ports910 may include different amounts of ingress ports in differentembodiments. As shown, the ingress ports 910 can receive network datathat is external to the switching element 900, which is indicated asincoming packets in this example. The ingress ports 910 can also receivenetwork data (e.g., packets) within the switching element 900 from thedispatch port 930. When the ingress ports 910 receive network data, theingress ports 910 forwards the network data to the forwarding tables940.

The forwarding tables 940 conceptually represent a set of forwardingtables for routing and modifying network data received from the ingressports 910. In some embodiments, the forwarding tables 940 include a setof records (or rules) that instruct the switching element 900 to routeand/or modify network data and send the network data to the egress ports920 and/or the dispatch port 930 based on defined routing criteria. Asnoted above, examples of routing criteria include source media accesscontrol (MAC) address, destination MAC, packet type, source InternetProtocol (IP) address, destination IP address, source port, destinationport, and/or virtual local area network (VLAN) identifier, among otherrouting criteria. In some embodiments, the switching element 900 routesnetwork data to a particular egress port according to the routingcriteria.

The egress ports 920 conceptually represent a set of ports through whichthe switching element 900 sends network data out of the switchingelement 900. The egress ports 920 may include different amounts ofegress ports in different embodiments. In some embodiments, some or allof the egress ports 920 may overlap with some or all of the ingressports 910. For instance, in some such embodiments, the set of ports ofthe egress ports 920 is the same set of ports as the set of ports ofingress ports 910. As illustrated in FIG. 9 , the egress ports 920receive network data after the switching element 900 processes thenetwork data based on the forwarding tables 940. When the egress ports910 receive network data (e.g., packets), the switching element 900sends the network data out of the egress ports 920, which is indicatedas outgoing packets in this example, based on the routing criteria inthe forwarding tables 940.

In some embodiments, the dispatch port 930 allows packets to bereprocessed by the forwarding tables 940. In some cases, the forwardingtables 940 are implemented as a single table (e.g., due to the switchingelement 900 s hardware and/or software limitations). However, someembodiments of the forwarding tables 940 may logically need more thanone table. Therefore, in order to implement multiple forwarding tablesin a single table, the dispatch port 930 may be used. For example, whenthe forwarding tables 940 processes a packet, the packet may be tagged(e.g., modifying a context tag of the packet or a header field of thepacket) and sent to the dispatch port 930 for the forwarding tables 940to process again. Based on the tag, the forwarding tables 940 processesthe packet using a different set of records. So logically, a differentforwarding table is processing the packet.

The dispatch port 930 receives after the switching element 900 processesthe network data according to the forwarding tables 940. As noted above,the switching element 900 might route the network data to the dispatchport 930 according to routing criteria defined the forwarding tables940. When the dispatch port 930 receives network data, the dispatch port930 sends the network data to the ingress ports 910 to be furtherprocessed by the forwarding tables 940. For example, the switchingelement 900 might modify the network data based on the forwarding tables940 and send the modified network data to the dispatch port 930 forfurther processing by the forwarding tables 940.

FIG. 10 conceptually illustrates an architectural diagram of a hardwareswitching element 1000 of some embodiments. As illustrated in thisfigure, the switching element 1000 includes ingress ports 1010, egressports 1020, dispatch port 1030, forwarding tables 1040, managementprocessor 1050, configuration database 1060, control plane 1070,communication interface 1080, and packet processor 1090.

The ingress ports 1010 are similar to the ingress ports 910 illustratedin FIG. 9 except the ingress ports 1010 send network data to the packetprocessor 1090 instead of forwarding tables. The egress ports 1020 aresimilar to the ingress ports 1020 illustrated in FIG. 07 except theegress ports 1020 receive network data from the packet processor 1090instead of forwarding tables. Similarly, the dispatch port 1030 issimilar to the dispatch port 930 of FIG. 9 except the dispatch port 1030receives network data from the packet processor 1090 instead offorwarding tables.

The management processor 1050 controls the operations and functions ofthe switching element 1000. As shown in FIG. 10 , the managementprocessor 1050 of some embodiments receives commands for controlling theswitching element 1000 through a switching control protocol. One exampleof a switching control protocol is the Openflow protocol. The Openflowprotocol, in some embodiments, is a communication protocol forcontrolling the forwarding plane (e.g., forwarding tables) of aswitching element. For instance, the Openflow protocol provides commandsfor adding flow entries to, removing flow entries from, and modifyingflow entries in the switching element 1000.

The management processor 1050 also receives configuration informationthrough a configuration protocol. When the management processor 1050receives configuration information, the management processor 1050 sendsthe configuration information to the configuration database 1060 for theconfiguration database 1060 to store. In some embodiments, configurationinformation includes information for configuring the switching element1000, such as information for configuring ingress ports, egress ports,QoS configurations for ports, etc.

When the management processor 1050 of some embodiments receivesswitching control commands and the configuration commands, themanagement processor 1050 translates such commands into equivalentcommands for configuring the switching element 1000 to implement thefunctionalities of the commands. For instance, when the managementprocessor 1050 receives a command to add a flow entry, the managementprocessor 1050 translates the flow entry into equivalent commands thatconfigure the switching element 1000 to perform functions equivalent tothe flow entry. In some embodiments, the management processor 1050 mightrequest configuration information from the configuration database 1060in order to perform translation operations.

Some embodiments of the management processor 1050 are implemented aselectronic circuitry while other embodiments of the management processor1050 are implemented as an embedded central processing unit (CPU) thatexecutes switching element management software (e.g., OVS) that performssome or all of the functions described above.

The configuration database 1060 of some embodiments stores configurationinformation that the configuration database 1060 receives from themanagement processor 1050. In addition, when the management processor1050 sends requests for configuration information to the configurationdatabase 1060, the configuration database 1060 retrieves the appropriateconfiguration information and sends the requested configurationinformation to the management processor 1050.

In some embodiments, the control plane 1070 stores a set of flow tablesthat each includes a set of flow entries (also referred to collectivelyas configured flow entries). The control plane 1070 of some embodimentsreceives flow entries from the management processor 1050 to add to theset of flow tables, and receives requests from the management processor1050 to remove and modify flow entries in the set of flow tables. Inaddition, some embodiments of the control plane 1070 might receiverequests from the management processor 1050 for flow tables and/or flowentries. In such instances, the control plane 1070 retrieves therequested flow tables and/or flow entries and sends the flow tablesand/or flow entries to the management processor 1050.

In addition, the control plane 1070 of some embodiments stores differentflow tables and/or flow entries that serve different purposes. Forinstance, as mentioned above, a switching element may be one of severalswitching elements in a network across which multiple logical switchingelements are implemented. In some such embodiments, the control plane1070 stores flow tables and/or flow entries for operating in thephysical domain (i.e., physical context) and stores flow tables and/orflow entries for operating in the logical domain (i.e., logicalcontext). In other words, the control plane 1070 of these embodimentsstores flow tables and/or flow entries for processing network data(e.g., packets) through logical switching elements and flow tablesand/or flow entries for processing network the data through physicalswitching elements in order to implement the logical switching elements.In this manner, the control plane 1070 allows the switching element 1000to facilitate implementing logical switching elements across theswitching element 1000 (and other switching elements in the managednetwork).

In some embodiments, the flow tables and/or flow entries for operatingin the physical domain process packets based on a set of fields in thepackets' header (e.g., source MAC address, destination MAC address,source IP address, destination IP address, source port number,destination port number) and the flow tables and/or flow entries foroperating in the logical domain process packets based on the packets'logical context ID (e.g., as described above by reference to FIG. 8 ) ora logical context tag (e.g., as described below by reference to FIGS.14, 15, 40, 41, and 44 ).

Some embodiments of the communication interface 1080 facilitatecommunication between management processor 1050 and packet processor1090. For instance, when the communication interface 1080 receivesmessages (e.g., commands) from the management processor 1050, thecommunication interface 1080 forwards the messages to the packetprocessor 1090 and when the communication interface 1080 receivesmessages from the packet processor 1090, the communication interface1080 forwards the messages to the management processor 1050. In someembodiments, the communication interface 1080 translates the messagessuch that the recipient of the message can understand the message beforesending the message to the recipient. The communication interface 1080can be implemented as a peripheral component interconnect (PCI) or PCIexpress bus in some embodiments. However, the communication interface1080 may be implemented as other types of busses in other embodiments.

In some embodiments, the forwarding tables 1040 store active flow tablesand/or flow entries that are used to determine operations for routing ormodifying network data (e.g., packets). In some embodiments, activetables and/or flow entries are a subset of the flow tables and/orentries stored in the control plane 1070 that the forwarding tables 1040is currently using or was recently using to process and route networkdata.

In this example, each flow entry is includes a qualifier and an action.The qualifier defines a set of fields to match against the network data.Examples of fields for matching network data include ingress port,source MAC address, destination MAC address, Ethernet type, VLAN ID,VLAN priority, multiprotocol label switching (MPLS) label, MPLS trafficclass, source IP address, destination IP address, transport controlprotocol (TCP)/user datagram protocol (UDP)/stream control transmissionprotocol (SCTP) source port, and/or TCP/UDP/SCTP destination port. Othertypes of packet header fields are possible as well in other embodiments.The action of a flow entry defines operations for processing the networkdata when the network data matches the qualifier of the flow entry.Examples of actions include modify the network data and route thenetwork data to a particular port or ports. Other embodiments provideadditional and/or other actions to apply to the network data.

In some embodiments, the packet processor 1090 processes network data(e.g., packets) that the packet processor 1090 receives from the ingressports 1010. Specifically, the packet processor 1090 processes (e.g.,route, modify, etc.) the network data based on flow entries in theforwarding tables 1040. In order to process the network data, the packetprocessor 1090 accesses the flow entries in the forwarding tables 1040.As mentioned above, the forwarding tables 1040 include a subset of flowtables and/or flow entries stored in the control plane 1070. When thepacket processor 1090 needs a flow table and/or flow entries that is notin the forwarding tables 1040, the packet processor 1090 requests thedesired flow table and/or flow entries, which are stored in the controlplane 1070, from the management processor 1050 through the communicationinterface 1080.

Based on the flow entries in the forwarding tables 1040, the packetprocessor 1090 sends the network data to one or more ports of the egressports 1020 or the dispatch port 1030. In some embodiments, the networkdata may match multiple flow entries in the forwarding tables 1040. Insuch cases, the packet processor 1090 might process the network databased on the first flow entry that has a qualifier that matches thenetwork data.

In some embodiments, the packet processor 1090 is anapplication-specific integrated circuit (ASIC) that performs some or allof the functions described above. In other embodiments, the packetprocessor 1090 is an embedded CPU that executes packet processingsoftware that performs some or all of the functions described above.

Different embodiments of the switching element 1000 may implement thepacket processor 1090 and forwarding tables 1040 differently. Forinstance, in some embodiments, the packet processor 1090 and forwardingtables 1040 are implemented as a multi-stage processing pipeline. Inthese embodiments, each flow entry in the forwarding tables 1040 areimplemented as one or more operations along one or more stages of themulti-stage packet processing pipeline. As explained above, themanagement processor 1050 of some embodiments translates flow entriesinto equivalent commands that configure the switching element 1000 toperform functions equivalent to the flow entry. Accordingly, themanagement processor 1050 would configure the multi-stage packetprocessing pipeline to perform the functions equivalent to the flowentries in the forwarding tables.

FIG. 11 conceptually illustrates an architectural diagram of a physicalhost 1100 that includes a software switching element 1110 (e.g., an OVS)of some embodiments. The top portion of FIG. 11 illustrates the physicalhost 1100, which includes the software switching element 1110 and fourVMs 1120-1135. In some embodiments, the physical host 1100 is the sameor similar as the hosts that are running software switching elements inFIGS. 3-5 . Different embodiments of the physical host 1100 can be adesktop computer, a server computer, a laptop, or any other type ofcomputing device. The bottom portion of FIG. 11 illustrates the physicalhost 1100 in more detail. As shown, the physical host 1100 includesphysical ports 1140, a hypervisor 1145, patch ports 1150, the softwareswitching element 1110, patch ports 1155, and the VMs 1120-1135.

In some embodiments, the physical ports 1140 of the physical host 1100are a set of network interface controllers (NICs) that are for receivingnetwork data and sending network data outside the physical host 1100. Insome embodiments, the physical ports 1140 are a set of wireless NICs.The physical ports 1140 of other embodiments are a combination of NICsand wireless NICs.

The hypervisor 1145 (also referred to as a virtual machine monitor(VMM)) of some embodiments is a virtualization application that managesmultiple operating systems (e.g., VMs) on the physical host 1100. Thatis, the hypervisor 1145 provides a virtualization layer in which otheroperating systems can run with the appearance of full access to theunderlying system hardware (not shown) of the physical host 1100 exceptsuch access is actually under the control of the hypervisor 1145. Inthis example, the hypervisor 1145 manages the VMs 1120-1135 running onthe physical host 1100.

In some embodiments, the hypervisor 245 manages system resources, suchas memory, processors (or processing units), persistent storage, or anyother type of system resource, for each of the operating systems thatthe hypervisor 1145 manages. For this example, the hypervisor 1145manages the physical ports 1140, the network resources of the physicalhost 1100. In particular, the hypervisor 1145 manages and controlsnetwork data flowing through the physical ports 1140 and the patch ports1150 by, for example, mapping each port of the patch ports 1150 to acorresponding port of the physical ports 1140.

Different embodiments use different hypervisors. In some embodiments,the hypervisor 1145 is a Xen hypervisor is used while, in otherembodiments, the hypervisor 1145 is a VMware hypervisor. Otherhypervisors can be used in other embodiments.

The patch ports 1150 are a set of virtual ports (e.g., virtual networkinterfaces (VIFs)). To the software switching element 1110 and thehypervisor 1145, the patch ports 1150 appear and behave similar tophysical ports on a hardware switching element. For instance, thesoftware switching element 1110 and the hypervisor 1145 may send andreceive network data through the patch ports 1150. In some embodiments,the patch ports 1150 are provided by the hypervisor 1145 to the softwareswitching element 1110 while, in other embodiments, the patch ports 1150are provided by the software switching element 1110 to the hypervisor1145.

The patch ports 1155 are a set of virtual ports that are similar to thepatch ports 250. That is, to the software switching element 1110 and theVMs 1120-1135, the patch ports 1155 appear and behave similar tophysical ports on a hardware switching element. As such, the softwareswitching element 1110 and the VMs 1120-1135 may send and receivenetwork data through the patch ports 1155. In some embodiments, thepatch ports 1155 are provided by the software switching element 1110 tothe VMs 1120-1135 while, in other embodiments, the patch ports 1155 areprovided by the VMs 1120-1135 to the software switching element 1110.

As shown, the software switching element 1110 includes a control plane1160, a configuration database 1165, a forwarding plane 1170, andforwarding tables 1175. The control plane 1160 of some embodiments issimilar to the control plane 1070 of FIG. 10 in that the control plane1160 also stores configured flow entries (i.e., a set of flow tablesthat each includes a set of flow entries). Also, the configurationdatabase 1165 is similar to the configuration database 1060 of FIG. 10 .That is, the configuration database 1165 stores configurationinformation for configuring the software switching element 1110. (e.g.,information for configuring ingress ports, egress ports, QoSconfigurations for ports, etc.)

In some embodiments, the forwarding plane 1170 and the forwarding tables1175 performs functions similar to ones performed by packet processor1090 and the forwarding tables 1040 described above by reference to FIG.10 . The forwarding plane 1170 of some embodiments processes networkdata (e.g., packets) that the forwarding plane 1170 receives from thepatch ports 1150 and the patch ports 1155. In some embodiments, theforwarding plane 1170 processes the network data by accessing the flowentries in the forwarding tables 1175. When the forwarding plane 1170needs a flow table and/or flow entries that is not in the forwardingtables 1175, the forwarding plane 1170 of some embodiments requests thedesired flow table and/or flow entries from the control plane 1070.

Based on the flow entries in the forwarding tables 1175, the forwardingplane 1170 sends the network data to one or more ports of the patchports 1150 and/or one or more ports of the patch ports 1155. In someembodiments, the network data may match multiple flow entries in theforwarding tables 1175. In these instances, the forwarding plane 1170might process the network data based on the first flow entry that has aqualifier that matches the network data.

FIG. 12 conceptually illustrates an architectural diagram of a softwareswitching element of some embodiments that is implemented in a host1200. In this example, the software switching element includes threecomponents—an OVS kernel module 1245, which runs in the kernel of the VM1285, and an OVS daemon 1265 and an OVS database (DB) daemon 1267, whichrun in the user space of the VM 1285. While FIG. 12 illustrates thesoftware switching elements as two components for the purpose ofexplanation, the OVS kernel module 1245, the OVS daemon 1265, and theOVS DB daemon 1267 collectively form the software switching elementrunning on the VM 1285. Accordingly, the OVS kernel module 1245, the OVSdaemon 1265, and the OVS DB daemon 1267 may be referred to as thesoftware switching element and/or the OVS switching element in thedescription of FIG. 12 . In some embodiments, the software switchingelement can be any of the software switching elements illustrated inFIGS. 3-5 and, in such cases, the host 1200 is the host in the rack ofhosts in which the software switching element is running.

As illustrated in FIG. 12 , the host 1200 includes hardware 1205,hypervisor 1220, and VMs 1285-1295. The hardware 1205 may includetypical computer hardware, such as processing units, volatile memory(e.g., random access memory (RAM)), non-volatile memory (e.g., hard discdrives, optical discs, etc.), network adapters, video adapters, or anyother type of computer hardware. As shown, the hardware 1205 includesNICs 1210 and 1215, which are typical network interface controllers forconnecting a computing device to a network.

The hypervisor 1220 is a software abstraction layer that runs on top ofthe hardware 1205 and runs below any operation system. The hypervisor1205 handles various management tasks, such as memory management,processor scheduling, or any other operations for controlling theexecution of the VMs 1285-1295. Moreover, the hypervisor 1220communicates with the VM 1285 to achieve various operations (e.g.,setting priorities). In some embodiments, the hypervisor 1220 is a Xenhypervisor while, in other embodiments, the hypervisor 1220 may be anyother type of hypervisor for providing hardware virtualization of thehardware 1205 on the host 1200.

As shown, the hypervisor 1220 includes device drivers 1225 and 1230 forthe NICs 1210 and 1215, respectively. The device drivers 1225 and 1230allow an operating system to interact with the hardware of the host1200. In this example, the device driver 1225 allows the VM 1285 tointeract with the NIC 1210. And the device driver 1230 allows the VM1285 to interact with the NIC 1215. The hypervisor 1220 may includeother device drivers (not shown) for allowing the VM 1285 to interactwith other hardware (not shown) in the host 1200.

VMs 1285-1295 are virtual machines running on the hypervisor 1220. Assuch, the VMs 1285-1295 run any number of different operating systems.Examples of such operations systems include Solaris, FreeBSD, or anyother type of Unix-based operating system. Other examples includeWindows-based operating systems as well.

In some embodiments, the VM 1285 is a unique virtual machine, whichincludes a modified Linux kernel, running on the hypervisor 1220. Insuch cases, the VM 1285 may be referred to as domain 0 or dom0 in someembodiments. The VM 1285 of such embodiments is responsible for managingand controlling other VMs running on the hypervisor 1220 (e.g., VMs 1290and 1295). For instance, the VM 1285 may have special rights to accessthe hardware 1205 of the host 1200. In such embodiments, other VMsrunning on the hypervisor 1220 interact with the VM 1285 in order toaccess the hardware 1205. In addition, the VM 1285 may be responsiblefor starting and stopping VMs on the hypervisor 1220. The VM 1285 mayperform other functions for managing and controlling the VMs running onthe hypervisor 1220.

Some embodiments of the VM 1285 may include several daemons (e.g., Linuxdaemons) for supporting the management and control of other VMs runningon the hypervisor 1220. Since the VM 1285 of some embodiments is managesand controls other VMs running on the hypervisor 1220, the VM 1285 maybe required to run on the hypervisor 1220 before any other VM is run onthe hypervisor 1220.

As shown in FIG. 12 , the VM 1285 includes a kernel and a user space. Insome embodiments, the kernel is the most basic component of an operatingsystem that runs on a separate memory space and is responsible formanaging system resources (e.g., communication between hardware andsoftware resources). In contrast, the user space is a memory space whereall user mode applications may run.

As shown, the user space of the VM 1285 includes the OVS daemon 1265 andthe OVS DB daemon 1267. Other applications (not shown) may be includedin the user space of the VM 1285 as well. The OVS daemon 1265 is anapplication that runs in the background of the user space of the VM1285. Some embodiments of the OVS daemon 1265 communicate with a networkcontroller 1280 in order to process and route packets that the VM 1285receives. For example, the OVS daemon 1265 receives commands from thenetwork controller 1280 regarding operations for processing and routingpackets that the VM 1285 receives. The OVS daemon 1265 communicates withthe network controller 1280 through the Openflow protocol. In someembodiments, another type of communication protocol is used.Additionally, some embodiments of the OVS daemon 1265 receivesconfiguration information from the OVS DB daemon 1267 to facilitate theprocessing and routing of packets.

In some embodiments, the OVS DB daemon 1267 is also an application thatruns in the background of the user space of the VM 1285. The OVS DBdaemon 1267 of some embodiments communicates with the network controller1280 in order to configure the OVS switching element (e.g., the OVSdaemon 1265 and/or the OVS kernel module 1245). For instance, the OVS DBdaemon 1267 receives configuration information from the networkcontroller 1280 for configuring ingress ports, egress ports, QoSconfigurations for ports, etc., and stores the configuration informationin a set of databases. In some embodiments, the OVS DB daemon 1267communicates with the network controller 1280 through a databasecommunication protocol (e.g., a JavaScript Object Notation (JSON) remoteprocedure call (RPC)-based protocol). In some embodiments, another typeof communication protocol is utilized. In some cases, the OVS DB daemon1267 may receive requests for configuration information from the OVSdaemon 1265. The OVS DB daemon 1267, in these cases, retrieves therequested configuration information (e.g., from a set of databases) andsends the configuration information to the OVS daemon 1265.

The network controller 1280 is similar to the various networkcontrollers described in this application, such as the ones described byreference to FIGS. 1-5 . That is, the network controller 1280 managesand controls the software switching element running on the VM 1285 ofthe host 1200.

FIG. 12 also illustrates that the OVS daemon 1265 includes an Openflowprotocol module 1270 and a flow processor 1275. The Openflow protocolmodule 1270 communicates with the network controller 1280 through theOpenflow protocol. For example, the Openflow protocol module 1270receives configuration information from the network controller 1280 forconfiguring the software switching element. Configuration informationmay include flows that specify rules (e.g. flow entries) for processingand routing packets. When the Openflow protocol module 1270 receivesconfiguration information from the network controller 1280, the Openflowprotocol module 1270 may translate the configuration information intoinformation that the flow processor 1275 can understand. In someembodiments, the Openflow protocol module 1270 is a library that the OVSdaemon 1265 accesses for some or all of the functions described above.

The flow processor 1275 manages the rules for processing and routingpackets. For instance, the flow processor 1275 stores rules (e.g., in astorage medium, such as a disc drive) that the flow processor 1275receives from the Openflow protocol module 1270 (which, in some cases,the Openflow protocol module 1270 receives from the network controller1280). In some embodiments, the rules are stored as a set of flow tablesthat each includes a set of flow entries (also referred to collectivelyas configured flow entries). As noted above, flow entries specifyoperations for processing and/or routing network data (e.g., packets)based on routing criteria. In addition, when the flow processor 1275receives commands from the Openflow protocol module 1270 to removerules, the flow processor 1275 removes the rules.

In some embodiments, the flow processor 1275 supports different types ofrules. For example, the flow processor 1275 of such embodiments supportswildcard rules and exact match rules. In some embodiments, an exactmatch rule is defined to match against every possible field of aparticular set of protocol stacks. A wildcard rule is defined to matchagainst a subset of the possible fields of the particular set ofprotocol stacks. As such, different exact match rules and wildcard rulesmay be defined for different set of protocol stacks.

The flow processor 1275 handles packets for which integration bridge1250 does not have a matching rule. For example, the flow processor 1275receives packets from the integration bridge 1250 that does not matchany of the rules stored in the integration bridge 1250. In such cases,the flow processor 1275 matches the packets against the rules stored inthe flow processor 1275, which include wildcard rules as well as exactmatch rules. When a packet matches an exact match rule or a wildcardrule, the flow processor 1275 sends the exact match rule or the wildcardrule and the packet to the integration bridge 1250 for the integrationbridge 1250 to process.

In some embodiment, when a packet matches a wildcard rule, the flowprocessor 1275 generates an exact match rule based on the wildcard ruleto which the packet matches. As mentioned above, a rule, in someembodiments, specifies an action to perform based on a qualifier. Assuch, in some embodiments, the generated exact match rule includes thecorresponding action specified in the wildcard rule from which the exactmatch rule is generated.

In other embodiment, when a packet matches a wildcard rule, the flowprocessor 1275 generates a wildcard rule that is more specific than thewildcard rule to which the packet matches. Thus, in some embodiments,the generated (and more specific) wildcard rule includes thecorresponding action specified in the wildcard rule from which the exactmatch rule is generated.

In some embodiments, the flow processor 1275 may not have a rule towhich the packet matches. In such cases, some embodiments of the flowprocess 1275 send the packet to the network controller 1280 (through theOpenflow protocol module 1270). However, in other cases, the flowprocessor 1275 may have received from the network controller 1280 acatchall rule that drops the packet when a rule to which the packetmatches does not exist in the flow processor 1275.

After the flow processor 1275 generates the exact match rule based onthe wildcard rule to which the packet originally matched, the flowprocessor 1275 sends the generated exact match rule and the packet tothe integration bridge 1250 for the integration bridge 1250 to process.This way, when the integration bridge 1250 receives a similar packetthat matches generated the exact match rule, the packet will be matchedagainst the generated exact match rule in the integration bridge 1250 sothe flow processor 1275 does not have to process the packet.

Some embodiments of the flow processor 1275 support rule priorities forspecifying the priority for a rule with respect to other rules. Forexample, when the flow processor 1275 matches a packet against the rulesstored in the flow processor 1275, the packet may match more than onerule. In these cases, rule priorities may be used to specify which ruleamong the rules to which the packet matches that is to be used to matchthe packet.

The flow processor 1275 of some embodiments is also responsible formanaging rules in the integration bridge 1250. As explained in furtherdetail below, the integration bridge 1250 of some embodiments storesonly active rules. In these embodiments, the flow processor 1275monitors the rules stored in the integration bridge 1250 and removes theactive rules that have not been access for a defined amount of time(e.g., 1 second, 3 seconds, 5, seconds, 10 seconds, etc.). In thismanner, the flow processor 1275 manages the integration bridge 1250 sothat the integration bridge 1250 stores rules that are being used orhave recently been used.

Although FIG. 12 illustrates one integration bridge, the OVS kernelmodule 1245 may include multiple integration bridges. For instance, insome embodiments, the OVS kernel module 1245 includes an integrationbridge for each logical switching element that is implemented across amanaged network to which the software switching element belongs. Thatis, the OVS kernel module 1245 has a corresponding integration bridgefor each logical switching element that is implemented across themanaged network.

As illustrated in FIG. 12 , the kernel includes a hypervisor networkstack 1240 and an OVS kernel module 1245. The hypervisor network stack1240 is an Internet Protocol (IP) network stack that runs on the VM1285. The hypervisor network stack 1240 processes and routes IP packetsthat are received from the OVS kernel module 1245 and the PIF bridges1255 and 1260. When processing a packet that is destined for a networkhost external to the host 1200, the hypervisor network stack 1240determines to which of physical interface (PIF) bridges 1255 and 1260the packet is to be sent. The hypervisor network stack 1240 may makesuch determination by examining the destination IP address of the packetand a set of routing tables (not shown). In some embodiments, thehypervisor network stack 1240 is provided by the hypervisor 1220.

The OVS kernel module 1245 processes and routes network data (e.g.,packets) between VMs running on the host 1200 and network hosts externalto the host 1200 (i.e., network data received through the NICs 1210 and1215). For example, the OVS kernel module 1245 of some embodimentsroutes packets between VMs running on the host 1200 and network hostsexternal to the host 1200 (e.g., when packets are not routed through atunnel) through a set of patch ports (not shown) that couple the OVSkernel module 1245 to the PIF bridges 1255 and 1260. In several of thefigures in this application (e.g., FIG. 11 ), forwarding tables areillustrated as part of a forwarding plane of a software switchingelement. However, the forwarding tables may be conceptualrepresentations and may be implemented by the OVS kernel module 1245, insome embodiments.

To facilitate the processing and routing of network data, the OVS kernelmodule 1245 communicates with OVS daemon 1265. For example, the OVSkernel module 1245 receives processing and routing information (e.g.,flow entries) from the OVS daemon 1265 that specifies how the OVS kernelmodule 1245 is to process and route packets when the OVS kernel module1245 receives packets. Some embodiments of the OVS kernel module 1245include a bridge interface (not shown) that allows the hypervisornetwork stack 1240 to send packets to and receiving packets from the OVSkernel module 1245. In other embodiments, the hypervisor 1240 sendspackets to and receives packets from the bridges included in OVS kernelmodule 1245 (e.g., integration bridge 1250 and/or PIF bridges 1255 and1260).

FIG. 12 illustrates that the OVS kernel module 1245 includes anintegration bridge 1250 and the PIF bridges 1255 and 1260. Theintegration bridge 1250 processes and routes packets received from thehypervisor network stack 1240, the VMs 1290 and 1295 (e.g., throughVIFs), and the PIF bridges 1255 and 1260. In some embodiments, a set ofpatch ports is directly connects two bridges. The integration bridge1250 of some such embodiments is directly coupled to each of the PIFbridges 1255 and 1260 through a set of patch ports. In some embodiments,the integration bridge 1250 receives packets from the hypervisor networkstack 1240 through a default hypervisor bridge (not shown) that handlespacket processing and routing. However, in such embodiments, a functionpointer (also referred to as a bridge hook) that instructs thehypervisor bridge to pass packets to the integration bridge 1250 isregistered with the hypervisor bridge.

In some embodiments, the set of rules that the integration bridge 1250stores are only exact match rules. The integration bridge 1250 of somesuch embodiments stores only active exact match rules, which are asubset of the rules stored in the flow processor 1275 (and/or rulesderived from rules stored in the flow processor 1275) that theintegration bridge 1250 is currently using or was recently using toprocess and route packets. The integration bridge 1250 of someembodiments stores a set of rules (e.g., flow entries) for performingmapping lookups and logical forwarding lookups, such as the onesdescribed below in further detail by reference to FIGS. 14, 40, 41, 42,and 43 . Some embodiments of the integration bridge 1250 may alsoperform standard layer 2 packet learning and routing.

In some embodiments, the OVS kernel module 1245 includes a PIF bridgefor each NIC in the hardware 1205. For instance, if the hardware 1205includes four NICs, the OVS kernel module 1245 would include four PIFbridges for each of the four NICs in the hardware 1205. In otherembodiments, a PIF bridge in the OVS kernel module 1245 may interactwith more than one NIC in the hardware 1205.

The PIF bridges 1255 and 1260 route network data between the hypervisornetwork stack 1240 and network hosts external to the host 1200 (i.e.,network data received through the NICs 1210 and 1215). As shown, the PIFbridge 1255 routes network data between the hypervisor network stack1240 and the NIC 1210 and the PIF bridge 1260 routes network databetween the hypervisor network stack 1240 and the NIC 1215. The PIFbridges 1255 and 1260 of some embodiments perform standard layer 2packet learning and routing. In some embodiments, the PIF bridges 1255and 1260 performs physical lookups/mapping, such as the ones describedbelow in further detail by reference to FIGS. 14, 40, 42, and 43 .

In some embodiments, the VM 1285 provides and controls the PIF bridges1255 and 1260. However, the network controller 1280 may, in someembodiments, control the PIF bridges 1255 and 1260 (via the OVS daemon1265) in order to implement various functionalities (e.g., quality ofservice (QoS)) of the software switching element.

In several of the figures in this application (e.g., FIG. 11 ),forwarding tables are illustrated as part of a forwarding plane of asoftware switching element. However, these forwarding tables may be, insome embodiments, conceptual representations that can be implemented bythe OVS kernel module 1245. Also, some of the figures in thisapplication (e.g., FIGS. 10, 11, and 13 ) illustrate a control plane ina switching element. These control planes may similarly be conceptualrepresentations, which can be implemented by the OVS daemon 1265, insome embodiments.

The architectural diagram of the software switching element and the hostillustrated in FIG. 12 is one exemplary configuration. One of ordinaryskill in the art will recognize that other configurations are possible.For instance, some embodiments may include several integration bridgesin the OVS kernel module, additional NICs and corresponding PIF bridges,and additional VMs.

The following will describe an exemplary operation of the OVS switchingelement illustrated in FIG. 12 according to some embodiments of theinvention. Specifically, a packet processing operation performed by theOVS switching element will be described. As described above, the OVSkernel module 1245 processes packets and routes packets. The OVS kernelmodule 1245 can receive packets in different ways. For instance, the OVSkernel module 1245 can receive a packet from the VM 1290 or the VM 1295through the VM's VIF. In particular, the OVS kernel module 1245 receivesthe packet from the VM 1290 or the VM 1295 at the integration bridge1250.

Furthermore, the OVS kernel module 1245 can receive a packet from anetwork host external to the host 1200 through one of the NICs 1210 and1215, the NIC's corresponding PIF bridge (i.e., PIF bridge 1225 or PIFbridge 1230), and the hypervisor network stack 1240. The hypervisornetwork stack 1240 then sends the packets to the integration bridge 1250of the OVS kernel bridge 1245. In some cases, the packet is receivedfrom a network host external to the host 1200 through a tunnel. In someembodiments, the tunnel terminates at the hypervisor network stack 1240.Thus, when the hypervisor network stack 1240 receives the packet throughthe tunnel, the hypervisor network stack 1240 unwraps (i.e.,decapsulates) the tunnel header and determines, based on the tunnelinformation (e.g., tunnel ID), which integration bridge of the OVSkernel module 1245 to which to send the unwrapped packet. As mentionedabove, the OVS kernel module 1245 of some embodiments may include anintegration bridge for each logical switching element that isimplemented across the managed network to which the OVS switchingelement belongs. Accordingly, the hypervisor network stack 1240determines the logical switching element to which the tunnel belongs,identifies the integration bridge that corresponds to the determinedlogical switching element, and sends the packet to the identifiedintegration bridge.

In addition, the OVS kernel module 1245 can receive a packet from anetwork host external to the host 1200 through one of the NICs 1210 and1215, the NIC's corresponding PIF bridge (i.e., PIF bridge 1225 or PIFbridge 1230), and a set of patch ports (not shown) that couple the PIFbridge to the OVS kernel module 1245. As noted above, the OVS kernelmodule 1245 of some embodiments may include an integration bridge foreach logical switching element that is implemented across the managednetwork to which the OVS switching element belongs. Accordingly, theNIC's corresponding PIF bridge determines the logical switching elementto which the tunnel belongs, identifies the integration bridge thatcorresponds to the determined logical switching element, and sends thepacket to the identified integration bridge.

When the integration bridge 1250 receives a packet in any of the mannersdescribed above, the integration bridge 1250 processes the packet androutes the packet. As noted above, some embodiments of the integrationbridge 1250 stores only active exact match rules, which are a subset ofthe rules stored in the flow processor 1275 (and/or rules derived fromrules stored in the flow processor 1275) that the integration bridge1250 is currently using or was recently using to process and routepackets. The integration bridge 1250 performs a lookup based on a set offields in the packet's header (e.g., by applying a hash function to theset of fields). In some embodiments, the set of fields may include afield for storing metadata that describes the packet. If the lookupreturns a rule to which the packet matches, the integration bridge 1250performs the action (e.g., forward the packet, drop the packet,reprocess the packet, etc.) specified in the rule. However, if thelookup does not return a rule, the integration bridge 1250 sends thepacket to the flow processor 1275 to process.

As explained above, the flow processor 1275 handles packets for whichthe integration bridge 1250 does not have a matching rule. When the flowprocessor 1275 receives the packet from the integration bridge 1250, theflow processor 1275 matches the packet against the rules stored in theflow processor 1275, which include wildcard rules as well as exact matchrules. When a packet matches an exact match rule, the flow processor1275 sends the exact match rule and the packet to the integration bridge1250 for the integration bridge 1250 to process. When a packet matches awildcard rule, the flow processor 1275 generates an exact match rulebased on the wildcard rule to which the packet matches, and sends thegenerated exact match rule and the packet to the integration bridge 1250for the integration bridge 1250 to process.

Although FIG. 12 illustrates the VM 1285 as a virtual machine, differentembodiments may implement the VM 1285 differently. For example, someembodiments may implement the VM 1285 as part of the hypervisor 1220. Insuch embodiments, the VM 1285 performs the same or similar functions asthose described above with respect to the VM 1285.

FIG. 13 conceptually illustrates a network control system 1300 of someembodiments for managing a switching element 1320. Specifically, FIG. 13conceptually illustrates communication protocols that are employed inorder for a network controller 1310 to communicate with and control theswitching element 1320. Accordingly, the network control system 1300 maybe used to manage and control the switching element 1320 in order toimplement logical switching elements across the switching element andother switching elements, which belong to a network managed by thenetwork controller 1300.

The network controller 1310 is similar to the network controllersdescribed above by reference to FIGS. 2-5 except the network controller1310 communicates with the switching element 1320 through a databaseconnection and an Openflow connection. In some embodiments, a JavaScriptObject Notation (JSON) remote procedure call (RPC)-based protocol isused to establish the database connection and to communicate (e.g.,updating databases) through the database connection. In otherembodiments, any of the many known database connection and communicationmethods (e.g., Java DataBase Connectivity (JDBC) or Open DatabaseConnectivity (ODBC)) may be used. The Openflow connection uses theOpenflow protocol to establish a connection and facilitatecommunication.

In some embodiments, the switching element 1320 is a software switchingelement (e.g., the OVS switching element illustrated in FIGS. 11 and 12) while, in other embodiments, the switching element 1320 is a hardwareswitching elements (e.g., the switching element illustrated in FIG. 10). Therefore, even for a hardware switching element, OVS is executed onthe hardware switching element. For example, referring to FIG. 10 ,which illustrates a hardware switching element, some embodiments of themanagement processor 1050 are implemented as an embedded centralprocessing unit (CPU) that executes switching element managementsoftware. In this example, the switching element management software isOVS.

As shown, the switching element 1320 includes a user space daemon 1325and a forwarding plane 1355. The user space daemon 1325 includes an OVSconnection manager 1330, a configuration database controller 1335, aconfiguration database 1340, a control plane controller 1345, and acontrol plane 1350. The OVS connection manager 1330 manages theconnection between the network controller 1310 and the configurationdatabase controller 1335, and the connection between the networkcontroller 1310 and the control plane controller 1345 so thatcommunications received over a particular connection is routed to theappropriate controller.

In some embodiments, the OVS connection manager 1330 translates thecommands and/or messages into a format that the recipient canunderstand. For example, when the network controller 1310 sends acommand to the switching element 1320 through the database connection,the OVS connection manager 1330 may translate the command so that theconfiguration database controller 1335 can understand the command.Similarly, when the network controller 1310 sends a command to theswitching element 1320 through the Openflow connection, the OVSconnection manager 1330 may translate the command so that the controlplane controller 1345 can understand the command.

The configuration database controller 1340 of some embodiments managesthe configuration database 1340 and receives commands from the OVSconnection manager 1330 related to the configuration database 1340.Examples of commands include create a table, delete a table, create arecord in a table, modify (i.e., update) a record in a table, delete arecord in a table, among other types of database commands. When theconfiguration database controller 1335 receives a command from the OVSconnection manager 1330, the configuration database controller 1335performs the corresponding action to the configuration database 1340.

The configuration database 1335 is similar to the configuration database1060, which is described above by reference to FIG. 10 . That is, theconfiguration database 1335 stores configuration information forconfiguring the switching element 1320. (e.g., information forconfiguring ingress ports, egress ports, QoS configurations for ports,etc.).

Some embodiments of the control plane controller 1345 manage theOpenflow rules stored in the control plane 1350 and receives commandsfrom the OVS connection manager 1330 related to the control plane 1350.Examples of commands include add a rule, modify (i.e., update) a rule,delete a rule, or other types of Openflow commands. When theconfiguration database controller 1335 receives a command from the OVSconnection manager 1330, the configuration database controller 1335performs the command's corresponding action to the configurationdatabase 1340.

The control plane 1350 is similar to the control plane 1070, which isdescribed above by reference to FIG. 10 . Thus, the control plane 1350stores configured flow entries that are, in some embodiments, a set offlow tables that each includes a set of flow entries. In some of theseembodiments, the control plane 1350 also stores flow tables and/or flowentries for operating in the physical domain (i.e., physical context)and stores flow tables and/or flow entries for operating in the logicaldomain (i.e., logical context) in order to implement logical switchingelements. In addition, the control plane 1350 receives flow entries fromthe network controller 1310 (through the OVS connection manager 1330 andthe control plane controller 1345) to add to the configured flowentries, and receives requests from the network controller 1310 (throughthe OVS connection manager 1330 and the control plane controller 1345)to remove and modify the configured flow entries. The control plane 1350may manage the flow entries stored in the forwarding plane 1355 in asimilar manner that the flow processor 1275 manages rules in theintegration bridge 1250. For example, the control plane 1350 monitorsthe flow entries stored in the forwarding plane 1355 and removes theflow entries that have not been access for a defined amount of time(e.g., 1 second, 3 seconds, 5, seconds, 10 seconds, etc.) so that thecontrol plane 1355 stores flow entries that are being used or haverecently been used.

The forwarding plane 1355 is similar to the forwarding plane describedabove by reference to FIG. 11 . That is, the forwarding plane 1355processes and routes network data (e.g., packets). In some embodiments,the forwarding plane 1355 stores only active rules (e.g., flow entries)that specify operations for processing and routing packets. In someembodiments, the forwarding plane 1355 sends packets to the controlplane 1350 that the forwarding plane 1355 cannot process (e.g., theforwarding plane 1355 does not have a flow entry that matches thepackets). As mentioned above, the switching element 1320 of someembodiments is a software switching element. In these embodiments, theforwarding plane 1355 is implemented as a software forwarding plane,such as the software forwarding planes described above by reference toFIGS. 11 and 12 . Similarly, in some embodiments where the switchingelement 1320 is a hardware switching elements, the forwarding plane 1355is implemented, for example, as the hardware forwarding plane describedabove by reference to FIG. 10 .

FIG. 14 conceptually illustrates a processing pipeline 1400 of someembodiments for processing network data through a logical switchingelement. In particular, the processing pipeline 1400 includes fourstages 1410-1440 for processing a packet through a logical switchingelement that is implemented across a set of managed switching elementsin a managed network. In some embodiments, each managed switchingelement in the managed network that receives the packet performs theprocessing pipeline 1400 when the managed switching element receives thepacket.

In some embodiments, a packet includes a header and a payload. Theheader includes, in some embodiments, a set of fields that containsinformation used for routing the packet through a network. Switchingelements may determine switching decisions based on the contained in theheader and may, in some cases, modify some or all of the header fields.As explained above, some embodiments determine switching decisions basedon flow entries in the switching elements' forwarding tables.

In some embodiments, the processing pipeline 1400 may be implemented byflow entries in the managed switching elements in the network. Forinstance, some or all of the flow entries are defined such that thepacket is processed against the flow entries based on the logicalcontext tag in the packet's header. Therefore, in some of theseembodiments, the managed switching elements are configured (e.g., by anetwork controller illustrated in FIGS. 1-5 ) with such flow entries.

In the first stage 1410 of the processing pipeline 1400, a logicalcontext lookup is performed on a packet to determine the logical contextof the packet. In some embodiments, the first stage 1410 is performedwhen the logical switching element receives the packet (e.g., the packetis initially received by a managed switching element in the network thatimplements the logical switching element).

In some embodiments, a logical context represents the state of thepacket with respect to the logical switching element. For example, someembodiments of the logical context may specify the logical switchingelement to which the packet belongs, the logical port of the logicalswitching element through which the packet was received, the logicalport of the logical switching element through which the packet is to betransmitted, the stage of the logical forwarding plane of the logicalswitching element the packet is at, etc. Referring to FIG. 8 as anexample, the logical context of some embodiments for packets sent fromtenant A's machines specify that the packets are to be processedaccording to the logical switching element 880, which is defined fortenant A (rather than the logical switching element 890, which isdefined for tenant B).

Some embodiments determine the logical context of a packet based on thesource MAC address of the packet (i.e., the machine from which thepacket was sent). Some embodiments perform the logical context lookupbased on the source MAC address of the packet and the inport (i.e.,ingress port) of the packet (i.e., the port of the managed switchingelement through which the packet was received). Other embodiments mayuse other fields in the packet's header (e.g., MPLS header, VLAN id,etc.) for determining the logical context of the packet.

After the logical context of the packet is determined, some embodimentsstore the information that represents the determined logical context inone or more fields of the packet's header. These fields may also bereferred to as a logical context tag or a logical context ID.Furthermore, the logical context tag may coincide with one or more knownheader fields (e.g., the VLAN id field) in some embodiments. As such,these embodiments do not utilize the known header field or itsaccompanying features in the manner that the header field is defined tobe used.

In the second stage 1420 of the processing pipeline 1400, logicalforwarding lookups are performed on the packets to determine where toroute the packet based on the logical switching element (e.g., thelogical port of the logical switching element of which to send thepacket out) through which the packet is being processed. In someembodiment, the logical forwarding lookups include a logical ingress ACLlookup for determining access control when the logical switching elementreceives the packet, a logical L2 lookup for determining where to routethe packet through a layer 2 network, and a logical egress ACL lookupfor determining access control before the logical switching elementroutes the packet out of the logical switching element. Alternatively,or in conjunction with the logical L2 lookup, some embodiments of thelogical forwarding lookups include a logical L3 lookup for determiningwhere to route the packet through a layer three network. These logicallookups are performed based on the logical context tag of the packet insome of these embodiments.

In some embodiments, the result of the logical forwarding lookups mayinclude dropping the packet, forwarding the packet to one or morelogical egress ports of the logical switching element, or forwarding thepacket to a dispatch port of the logical switching element. When thelogical forwarding lookups determines that the packet is to be routed tothe dispatch port of the logical switching element, some embodimentsrepeat the logical forwarding lookups until the packet is determined tobe either dropped or forwarded to one or more logical egress ports.

Next, the third stage 1430 of the processing pipeline 1400 performs amapping lookup on the packet. In some embodiments, the mapping lookup isa logical to physical mapping lookup that determines the logical egressport of the logical switching element. That is, the mapping lookupdetermines one or more ports of one or more managed switching elementsthat correspond to the logical egress port of the logical switchingelement through which the packet is to be sent out. For instance, if thepacket is a broadcast packet or a multicast packet, the third stage 1430of some embodiments determines the ports of the managed switchingelements that correspond to the logical egress ports of the logicalswitching element through which the packet is to be broadcasted ormulticasted out (i.e., the logical ports to which the intendedrecipients of the packet is coupled). If the packet is a unicast packet,the third stage 1430 determines a port of a managed switching elementthat corresponds to the logical egress port of the logical switchingelement through which the packet is to be sent out (i.e., the logicalport to which the intended recipient of the packet is coupled). In someembodiments of the third stage 1430, the mapping lookups are performedbased on the logical context tag of the packet.

At the fourth stage 1440 of the processing pipeline 1400, a physicallookup is performed. The physical lookup of some embodiments determinesoperations for routing the packet to the physical port(s) thatcorresponds to the logical egress port(s) that was determined in thethird stage 1430. For example, the physical lookup of some embodimentsdetermines one or more ports of the managed switching element on whichthe processing pipeline 1400 is being performed through which to sendthe packet out in order for the packet to reach the physical port(s)determined in the third stage 1430. This way, the managed switchingelements can route the packet along the correct path in the network forthe packet to reach the determined physical port(s) that corresponds tothe logical egress port(s).

Some embodiments remove the logical context tag after the fourth stage1440 is completed in order to return the packet to its original statebefore the packet was processed by the processing pipeline 1400.

As mentioned above, in some embodiments, the processing pipeline 1400 isperformed by each managed switching element in the managed network thatis used to implement the logical switching element. In some embodiments,some of the managed switching elements perform only a portion of theprocessing pipeline 1400. For example, in some embodiments, the managedswitching element that initially receives the packet may perform thefirst-fourth stages 1410-1440 and the remaining managed switchingelements that subsequently receive the packet only perform the first,third, and fourth stages 1410, 1430, and 1440.

FIG. 15 conceptually illustrates a process 1500 of some embodiments forimplementing a processing pipeline, such as the processing pipeline1400, that is distributed across managed switching elements according toflow entries in the managed switching elements. In some embodiments, theprocess 1500 is performed by each managed switching element in a managednetwork in order to process a packet through a logical switching elementthat is implemented across the managed switching elements.

The process 1500 begins by determining (at 1505) whether the packet hasa logical context tag. When the process 1500 determines that the packetdoes not have a logical context tag, the process 1500 determines (at1510) whether the packet matches a flow entry that specifies a logicalcontext. In some embodiments, the process 1500 determines the packet'slogical context in a similar fashion as that described above byreference to the first stage 1410 of FIG. 14 . That is, the process 1500determines the logical context of the packet based on a defined set offields in the packet's header (e.g., the source MAC address, inport,etc.).

When the process 1500 determines that the packet does not match a flowentry that specifies a logical context, the process 1500 drops (at 1535)the packet and the process 1500 then ends. When the process 1500determines that the packet matches a flow entry that specifies a logicalcontext, the process 1500 adds (at 1515) a logical context tag to theheader of the packet. After the process 1500 adds the logical contexttag to the header of the packet, the process 1500 proceeds to 1520. Whenthe process 1500 determines that the packet does have a logical contexttag, the process 1500 proceeds to 1520.

At 1520, the process 1500 determines whether the packet matches a flowentry that specifies the packet's logical context tag to be modified. Insome embodiments, the flow entries that the process 1500 matches thepacket against are flow entries that implement the logical ingress ACLlookup described above by reference to the second stage 1420 of FIG. 14. When the process 1500 determines that the packet matches a flow entrythat specifies the packet's logical context tag to be modified, theprocess 1500 modifies (at 1525) the packet according to the flow entryagainst which the packet matches. Then, the process 1500 proceeds to1530. When the process 1500 determines that the packet does not match aflow entry that specifies the packet's logical context tag to bemodified, the process 1500 proceeds to 1530.

Next, the process 1500 determines (at 1530) whether the packet matches aflow entry that specifies the packet to be dropped. In some embodiments,the flow entries that the process 1500 matches the packet against areflow entries that implement the logical L2 lookup described above byreference to the second stage 1420 of FIG. 14 . When the process 1500determines that the packet matches a flow entry that specifies thepacket to be dropped, the process 1500 drops (at 1535) the packet andthe process 1500 ends.

When the process 1500 determines that the packet does not match a flowentry that specifies the packet to be dropped, the process 1500determines (at 1540) whether the packet matches a flow entry thatspecifies the destination of the packet is local. In some embodiments,the destination of the packet is local when the recipient of the packetis coupled to the managed switching element on which the process 1500 isbeing performed. When the process 1500 determines that the packetmatches a flow entry that specifies the destination of the packet islocal, the process 1500 removes (at 1545) the logical context tag fromthe packet's header. Next, the process 1500 forwards (at 1550) thepacket to the local destination. In some embodiments, the process 1500determines the local destination by matching the packet against flowentries that implement the logical L2 lookup described above byreference to the second stage 1420 of FIG. 14 . After forwarding thepacket to the local destination, the process 1500 ends.

When the process 1500 determines that the packet does not match a flowentry that specifies the destination of the packet is local, the process1500 forwards (at 1555) the packet to the next managed switching elementfor further processing. Then, the process 1500 ends.

III. Hierarchical Switching Architecture

FIG. 16 conceptually illustrates a network architecture 1600 of someembodiments that includes a pool node 1605. The network architecture1600 is similar to the network architecture 100 illustrated in FIG. 1 ,but the network architecture 1600 also includes the pool node 1605 andthe managed switching element 130 is no longer connected to the managedswitching element 140. For purposes of explanation and simplicity, thenetwork controllers 110 and 120 are not shown in FIG. 16 . In addition,the machines 155, 160, 170, and 175 are indicated as belonging to atenant A, and the machines 165, 180, and 185 are indicated as belongingto a tenant B.

In some embodiments, the pool node 1605 is a switching element (e.g., ahardware switching element or an OVS) that is coupled to and positionedabove the managed switching elements 130-150 in the hierarchy of thenetwork architecture 1600 to assist in the implementation of logicalswitching elements across the managed switching elements 130-150. Thefollowing will describe some of the functions that some embodiments ofthe pool node 1605 provide.

The pool node 1605 of some embodiments is responsible for processingpackets that the managed switching elements 130-150 cannot process. Ininstances where one of the managed switching elements 130-150 cannotprocess a packet, the managed switching element sends the packet to thepool node 1605 to process. For instance, the pool nodes 1605 processespackets with destination MAC addresses that are not known to one of themanaged switching elements 130-150 (e.g., the managed switching elementdoes not have a flow entry that matches the destination MAC address). Insome cases, one of the managed switching elements 130-150 cannot processa packet due to the limited storage capacity of the managed switchingelement and does not include flow entries for processing the packet.Another example where the managed switching elements 130-150 cannotprocess a packet is because the packet is destined for a remote networkthat may not be managed by the network controllers 110 and 120.

In some embodiments, the pool node 1605 serves as a communication bridgebetween managed switching elements. Referring to FIG. 16 as an example,absent the pool node 1605, the managed switching element 130 cannotcommunicate with the managed switching elements 140 and 150. Therefore,when the managed switching element 130 wants to send packets, forexample, to the managed switching element 140 or the managed switchingelement 150, the managed switching element 130 sends the packets to thepool node 1605 to forward to the managed switching element 140 or themanaged switching element 150. Similarly, when the managed switchingelement 140 or the managed switching element 150 wants to send packetsto the managed switching element 130, the managed switching element 140or the managed switching element 150 sends the packets to the pool node1605 to forward to the managed switching element 130.

Some embodiments of the pool node 1605 process packets are that areintended for multiple recipients (e.g., broadcast packets and multicastpackets) in the same logical network. For instance, when one of themanaged switching elements 130-150 receives a broadcast or multicastpacket from one of the machines, the managed switching element sends thebroadcast or multicast packet to the pool node 1605 for processing.Referring to FIG. 16 as an example, when the managed switching element130 receives a broadcast from the machine 155, the managed switchingelement 130 sends the broadcast packet to the pool node 1605. The poolnode 1605 determines that the broadcast is destined for the machines ontenant A's logical network. Accordingly, the pool node 1605 determinesthat the machines 155, 160, 170, and 175 belong to tenant A and sendsthe packet to each of those machines. The pool node 1605 processesmulticast packets in a similar manner except, for the multicast packet,the pool node 1650 identifies the intended recipients of the multicastpacket.

As explained above, the pool node 1605 of some embodiments processespackets that are intended for multiple recipients in the same logicalnetwork. FIG. 17 conceptually illustrates an example multi-recipientpacket flow through the network architecture 1600 illustrated in FIG. 16according to some embodiments of the invention. Specifically, FIG. 17conceptually illustrates a managed switching element performing thereplication of packets for the multi-recipient packet.

In this example, tenant B's machine 165 sends a multi-recipient packet(e.g., a broadcast packet or a multicast packet) to the managedswitching element 130. In some embodiments, the multi-recipient packetspecifies a destination MAC address that is defined (e.g., by a networkcontroller managing) to indicate the packet is a multi-recipient packet.Some embodiments might indicate that the packet is a multi-recipientpacket through data stored in a set of fields (e.g., a context tag) inthe packet's header. The managed switching element 130 identifies thepacket as a multi-recipient packet based on the defined destination MACaddress and/or the set of header fields. Since the pool node 1605 isresponsible for processing multi-recipient packets, the managedswitching element 130 forwards the packet to the pool node 1605 forprocessing.

When the pool node 1605 receives the packet from the managed switchingelement 130, the pool node 1605 determines that the packet is amulti-recipient packet by examining the destination MAC address of thepacket and/or the set of header fields. In some embodiments, the packetalso specifies the logical network to which the packet belongs (e.g.,via a context tag). In this example, the packet specifies that thepacket belongs to the logical network that includes tenant B's machines(machines 165, 180, and 185 in this example). After the pool node 1605determines that logical network to which the packet belongs, the poolnode 1605 determines the managed switching elements to which to routethe multi-recipient packet. Since the managed switching element 140 isnot coupled to any of tenant B's machines, the pool node 1605 onlyforwards the multi-recipient packet to the managed switching element150.

When the managed switching element 150 receives the packet, the managedswitching element 150 determines that the packet is a multi-recipientpacket by examining the destination MAC address of the packet. Themanaged switching element 150 then determines the logical network towhich the packet belongs and identifies the machines coupled to themanaged switching element 150 that belong to the logical network towhich the packet belongs. For this example, the packet belongs to tenantB's logical network. Therefore, the managed switching element 150identifies the machines 180 and 185 as the machines coupled to themanaged switching element 150 that belong to tenant B's logical network.Then, the managed switching element 150 replicates the multi-recipientpacket for each identified machine, modifies each replicated packet tospecify the MAC address of the corresponding machine as the packet'sdestination MAC address, and sends the replicated packets to themachines.

As shown, FIG. 17 illustrates a packet flow of a multi-recipient packetthrough a network architecture of some embodiments where a managedswitching element performs the replication of packets for themulti-recipient packet. However, in some embodiments, the pool node ofsome embodiments may perform the replication of packets for amulti-recipient packet. FIG. 18 conceptually illustrates such an examplemulti-recipient packet flow through the network architecture 1600illustrated in FIG. 16 according to some embodiments of the invention.

For this example, tenant A's machine 175 sends a multi-recipient packet(e.g., a broadcast packet or a multicast packet) to the managedswitching element 150 that specifies tenant A's machine 155 and 160 asrecipients of the packet. In some embodiments, the multi-recipientpacket specifies a destination MAC address that is defined (e.g., by anetwork controller managing) to indicate the packet is a multi-recipientpacket and the recipients of the multi-recipient packet. Someembodiments might indicate that the packet is a multi-recipient packetthrough data stored in a set of fields (e.g., a context tag) in thepacket's header. The managed switching element 130 identifies the packetas a multi-recipient packet based on the defined destination MAC addressand/or the set of header fields. As the pool node 1605 is responsiblefor processing multi-recipient packets, the managed switching element150 forwards the packet to the pool node 1605 for processing.

When the pool node 1605 receives the packet from the managed switchingelement 150, the pool node 1605 determines that the packet is amulti-recipient packet by examining the destination MAC address of thepacket and/or the set of header fields. In some embodiments, the packetalso specifies the logical network to which the packet belongs (e.g.,via a context tag). In this example, the packet specifies that thepacket belongs to the logical network that includes tenant A's machines(machines 155, 160, 170, and 175 in this example). After the pool node1605 determines the logical network to which the packet belongs, thepool node 1605 identifies the set of managed switching elements (themanaged switching element 130 in this example) to which the intendedrecipients of the multi-recipient packet (the machines 155 and 160 inthis example) are coupled. The pool node 1605 then replicates themulti-recipient packet and sends a copy of the multi-recipient packet toeach of the identified set of managed switching elements.

The above description by reference to FIGS. 17 and 18 describes packetsthat are sent from a managed switching element to a pool node and from apool node to a managed switching element. In some embodiments, thepackets are sent through tunnels in a similar manner that is describedabove by reference to FIGS. 6 and 7 .

FIG. 19 conceptually illustrates an example of the pool node 1605configured to assist in processing packets for the managed switchingelements 130 and 150. In particular, this figure illustrates the managedswitching elements 130 and 150 configured (e.g., by a network controllerillustrated in FIGS. 1-5 ) with flow entries for processing packets andthe pool node 1605 configured (e.g., by a network controller illustratedin FIGS. 1-5 ) with flow entries for processing packets for the managedswitching elements 130 and 150.

As shown, the managed switching element 130 includes a forwarding table1920 and the managed switching element 150 includes a forwarding table1930. As noted above, the managed switching elements of some embodimentsmay have limited storage capacity and cannot store all the necessaryflow entries to process the different packets in the network. In thisexample, the managed switching element 130 can only store 27 flowentries (i.e., 9 flow entries for each of the machines 155-165) and themanaged switching element 150 can only store 21 flow entries (i.e., 7flow entries for each of the machines 175-185). The flow entries in eachof the forwarding tables 1920 and 1930 conceptually represent thepackets that the managed switching elements 130 and 150 can process.

As described above, the pool node 1605 processes packets that themanaged switching elements 130 and 150 cannot process (e.g., unknowndestination MAC address, broadcast and multicast packets, etc.). Asshown, the pool node 1605 includes a forwarding table 1910 with m+n flowentries. The flow entries in the forwarding table 1910 conceptuallyrepresent flow entries for processing packets that the managed switchingelements 130 and 150 cannot process.

In some embodiments, a pool node includes all the flow entries that areused to manage the network. For instance, referring to FIG. 19 as anexample, the pool node 1605 of such embodiments would include the flowentries in the forwarding tables 1920 and 1930 in addition to the flowentries shown in the forwarding table 1910. Moreover, a pool node ofsome embodiments includes information (e.g., MAC addresses) related toevery machine in the managed network. In some such embodiments, the poolnode would include flow entries for forwarding network data from everymachine in the managed network to each other. In cases where a managednetwork includes multiple pool nodes, some embodiments configure eachpool node similarly while other embodiments may configure one or morepool nodes differently.

Although FIG. 19 shows forwarding tables with the same number of flowentries for each machine stored in a forwarding table of the managedswitching elements and pool node, this figure illustrates an exemplaryconfiguration of the managed switching elements and the pool node. Oneof ordinary skill will recognize that the managed switching elements andthe pool node may include multiple forwarding tables with a differentnumber of flow entries for each of the different machines.

FIG. 20 conceptually illustrates a process 2000 of some embodiments forprocessing packets. In some embodiments, the process 2000 is performedby each managed switching element in a managed network. Specifically,the managed switching elements of some embodiments perform the process2000 when performing the second stage 1420 of the processing pipeline1400, which is described above by reference to FIG. 14 .

The process 2000 starts by determining (at 2010) whether the packet hasan unknown destination MAC address. In some embodiments, the destinationMAC address of the packet is unknown when the managed switching elementthat is performing the process 2000 does not have a flow entry thatmatches the packet's destination MAC address. When the process 2000determines that the packet does not have an unknown destination MACaddress, the process 2000 proceeds to 2020. Otherwise, the process 2000forwards (at 2060) the packet to a pool node and then the process 2000ends.

Next, the process 2000 determines (at 2020) whether the packet can beprocessed. In some embodiments, the packet can be processed when themanaged switching element on which the process 2000 is being performedhas a flow entry that matches the packet. When the process 2000determines that the packet cannot be processed, the process 2000forwards (at 2060) the packet to a pool node and then the process 2000ends.

When the process 2000 determines that the packet can be processed, theprocess 2000 processes (at 2030) the packet. The process 2000 of someembodiments processes the packet by performing the action specified inthe flow entry that matches the packet. After processing the packet, theprocess 2000 proceeds to 2040.

At 2040, the process 2000 determines whether the packet is a multicastor broadcast packet. Some embodiments define a multicast or broadcastpacket as a packet with defined values in a set of header fields (e.g.,destination MAC address, inport, etc.). When the process 2000 determinesthat the packet is not a multicast or broadcast packet, the process 2000ends. Otherwise, the process 2000 determines (at 2050) whether thepacket needs further processing. A packet may need further processingwhen the packet is a multicast or broadcast packet and one or more ofthe recipients of the multicast or broadcast packet are unknown (e.g.,the recipients are not coupled to the managed switching element that isperforming the process 2000).

When the process 2000 determines that the packet needs furtherprocessing, the process 2000 forwards (at 2060) the packet to a poolnode and then the process 2000 ends. When the process 2000 determinesthat the packet does not need further processing, the process 2000 ends.

In some embodiments, some or all of the operations in the process 2000is implemented by flow entries in the managed switching element on whichthe process 2000 is performed. For instance, the managed switchingelement may include a set of flow entries that define a broadcast ormulticast packet in some such embodiments. In such cases, the managedswitching element performs a lookup on the set of flow entries todetermine whether a packet is a broadcast or multicast packet (i.e.,whether the packet matches against the set of flow entries).

FIG. 21 conceptually illustrates a network architecture 2100 of someembodiments that includes root nodes 2105 and 2110. As shown, thenetwork architecture 2100 includes the root nodes 2105 and 2110, poolnodes 2115-2130, and managed switching elements 2135-2170. FIG. 21 alsoshows that each zone include a root node. In some embodiments, each zonein the network includes only one root node while, in other embodiments,each zone in the network can include several root nodes. In thisapplication, a root node may also be referred to as a root bridge.

In some embodiments, a root node is similar to a pool node in that theroot node is a switching element (e.g., a hardware switching element oran OVS) that is for assisting in the implementation of logical switchingelements across managed switching elements. However, the root nodeprovides different functions than a pool node and is positioned at adifferent level in the network hierarchy. The following will describesome functions that the root node of some embodiments provides.

Some embodiments of the root nodes 2105 and 2110 provide a communicationbridge between zones in the network. In some embodiments, a zone is adefined group of machines in a network. A zone may be defined any numberof different ways in different embodiments. For instance, a zone may bedefined as a group of machines in an office, a group of machines in asection of a data center, a group of machines in a building. As shown,zone 1 of the network architecture includes the pool nodes 2115 and 2120and the managed switching elements 2135-2150 and the zone 2 of thenetwork architecture includes the pool nodes 2125 and 2130 and themanaged switching elements 2155-2170.

As shown in FIG. 21 , the network elements in zone 1 of the networkcannot communicate with the network elements in zone 2 of the network.When a network element in one of the zones wants to communicate with anetwork element in the other zone, such communications are forwarded tothe corresponding root node in the zone. For instance, if the managedswitching element 2135 wants to send a packet to the managed switchingelement 2170, the managed switching element 2135 sends the packets tothe pool node 2115, which sends the packet to the root node 2105. Theroot node 2105 of zone 1 then forwards the packet to the root node 2110of zone 2 to forward to the managed switching element 2170 through thepool node 2130.

In some embodiments, the root nodes 2105 and 2110 perform logicalcontext learning. Logical context learning, in some embodiments, is aprocess of identifying the network element(s) to which packets areforwarded so that the packets can reach the packets' intendeddestination. Referring to FIG. 21 as an example, if the root node 2105receives from the pool node 2115 a packet from a new machine (e.g., thepacket includes an unknown source MAC address or IP address) that hasrecently been connected to the managed switching element 2135, the rootnode 2105 “learns” that the root node 2105 should forward packetsdestined for the new machine to the root node 2115 (as opposed toforwarding the packets to the pool node 2120 or the root node 2110). Byperforming logical context learning, the root nodes 2105 and 2110 ofsome embodiments is indirectly aware of the location of all the networkelements in the network and can thus forward packets to the correctnetwork element in order for packets to reach their intendeddestinations. Thus, when the pool nodes 2115-2130 do not know or cannotdetermine the logical context of a packet, the packet is sent to thecorresponding root node in the pool node's zone for processing (e.g., toforward to the packet's intended destination).

As described above, FIG. 21 shows root nodes as separate components atthe top of a network architecture hierarchy. However, in someembodiments, a similar network architecture may be implemented with poolnodes, which include some or all of the functions described above byreference to the root nodes in FIG. 21 , in place of root nodes at thetop of the network architecture hierarchy. In other embodiments, thesome or all of the root node functions are implemented by each of thepool nodes. In addition, while FIG. 21 illustrates one level of poolnodes in the hierarchy of a network architecture, different embodimentsof different network architectures may include different numbers oflevels of pool nodes in the hierarchy of the network architecture aswell as any number pool nodes at each level in the hierarchy of thenetwork architecture.

FIG. 22 conceptually illustrates an architectural diagram of a pool node2210 of some embodiments. In particular, FIG. 22 conceptuallyillustrates an example of a root node 2230 (i.e., root bridge) that isincluded in the pool node 2210. In some embodiments, the pool node 2210is general computing device (e.g., an x86 computing device) that runs anoperating system, such as a Unix-based operating system.

As shown, the pool node 2210 includes pool node network stack 2220, theroot bridge 2230, patch bridge 2240, and a set of NICs 2250. In someembodiments, each NIC in the set of NICs 2250 is typical networkinterface controllers for connecting a computing device to one or morenetworks and sending and receiving network data (e.g., packets) oversuch networks. In addition, the set of NICs 2250 sends and receivesnetwork data from the pool node network stack 2220.

The pool node network stack 2220 is similar to the hypervisor networkstack described above by reference to FIG. 12 . The pool node networkstack 2220 is an IP network stack that runs on the pool node 2210. Also,the pool node network stack 2220 processes and routes IP packets thatare received from the patch bridge 2240 and the set of NICs 2250, byutilizing a set of routing tables (not shown) to route the packets.

In some embodiments, the patch bridge 2240 stores a set of rules (e.g.,flow entries) that specify operations for processing and routingpackets. The patch bridge 2240 communicates with a network controller2260 in order to process and route packets that the patch bridge 2240receives. For instance, the patch bridge 2240 receives commands from thenetwork controller 2260 related to processing and routing of packetsthat the pool node 2210 receives. In some embodiments, the patch bridge2240 communicates with the network controller 2260 through the Openflowprotocol while, in other embodiments, another type of communicationprotocol may be used. The network controller 2260 is similar to thevarious network controllers described in this application, such as theones described by reference to FIGS. 1-5 . The network controller 2260manages and controls the switching element (OVS in this example) that isrunning on the pool node 2210.

As explained above, a pool node of some embodiments is responsible forprocessing packets that managed switching elements in a managed networkcannot process. In this example, the patch bridge 2240 processes androutes such packets. The patch bridge 2240 receives packets from managedswitching elements through the set of NICs 2250 and the pool nodenetwork stack 2220. When the patch bridge 2240 receives a packet, thepatch bridge 2240 processes and routes the packet according to the setof rules stored in the patch bridge 2240. In some cases, the patchbridge 2240 cannot process a packet (e.g., the patch bridge 2240 doesnot have a rule to which the packet matches). In these cases, the patchbridge 2240 sends the packet to the root bridge 2230 for processing.

Some embodiments of the root bridge 2230 are responsible for a learningfunction. The root bridge 2230 of some embodiments stores a set oftables of learned MAC addresses (unlike the pool nodes and managedswitches of some embodiments, which are controlled by a networkcontroller). The root bridge 2230 learns MAC addresses in the typicalmanner that layer 2 switches learn MAC addresses. For instance, when theroot bridge 2230 does not know a MAC address (i.e., a destination MACaddress of a packet is not included in the set of tables of learned MACaddresses), the root bridge 2230 floods all of the ports of the rootbridge 2230 and records the MAC address of the packet that responds tothe flood in the set of tables. As another example, when the root bride2230 receives a packet that includes a destination MAC address that theroot bridge 2230 does not know (i.e., the destination MAC address of thepacket is not included in the set of tables of learned MAC addresses),the root bridge 2230 records the source MAC address of the packet in theset of tables of learned MAC addresses. When the root bridge 2230 knowsthe MAC address of a packet (i.e., the MAC address is included in theset of tables of learned MAC addresses), the root bridge 2230 sends thepacket to the patch bridge 2240 to forward to the appropriate NIC in theset of NICs 2250 in order for the packet to reach the packet'sdestination. In some embodiments, the root bridge 2230 and the patchbridge 2240 communicate through a set of patch ports, which are forconnecting two bridges directly together. In some embodiments, the rootbridge 2230 may be directly connected to one or more extenders. In someof these embodiments, a tunnel is established between the root bridge2230 and each of the extenders in order for the root bridge 2230 and theextenders to communicate.

Although FIG. 22 illustrates a pool node that includes a root bridge,some embodiments may not include a root bridge. In some of theseembodiments, the functions described above are implemented in the patchbridge of the pool node.

FIG. 23 conceptually illustrates a network architecture 2300 of someembodiments that includes extenders 2305 and 2310. This figure shows thenetwork architecture 2300 that includes two managed networks, a SanDiego zone and a Chicago zone. In this example, the San Diego zone andthe Chicago zone are each controlled by a network controller (or controlclusters). As shown, the San Diego zone includes the extender 2305, arouter 2376, a root node 2320, pool nodes 2335 and 2340, and managedswitching elements 2352-2362, and the router 2376, the root node 2320,the pool nodes 2335 and 2340, and managed switching elements 2352-2362are physically located in a datacenter in San Diego. The Chicago zoneincludes the extender 2310, a router 2378, root nodes 2325 and 2330,pool nodes 2345 and 2350, and the managed switching elements 2364-2374.Also, the extenders 2305 and 2310, the router 2378, the root nodes 2325and 2330, the pool nodes 2345 and 2350, and the managed switchingelements 2364-2374 are physically located in a datacenter in Chicago.

In some embodiments, an extender is a switching element (e.g., ahardware switching element or an OVS) for communicatively bridgingremote managed networks that are separated by one or more othernetworks. As shown in FIG. 23 , the San Diego zone and the Chicago zoneare separated by external network 2315. To allow communication betweenthe two zones, the extender 2305, which is physically located in theChicago datacenter, and the extender 2310 provide a communication bridgebetween the San Diego zone and the Chicago zone. In this example, thecommunication bridge between the two zones is partially provided by atunnel, which is established using any of the tunneling protocolsdescribed above by reference to FIGS. 6 and 7 , between the extender2305 and the root node 2320. In addition, the tunnel in FIG. 23 is asecure tunnel that is secured using Internet Protocol Security (IPsec)since communications are sent between the two zones through the externalnetwork 2315, which may be unsecure.

The above FIG. 23 describes extenders that are used to bridge managednetworks that are separately by an external network. However, theextenders of some embodiments can be used to bridge a managed networkwith an unmanaged network. An unmanaged network is a network that is notmanaged by a network controller, in some embodiments. The following FIG.24 conceptually illustrates an example of extenders used for such apurpose.

FIG. 24 conceptually illustrates a network architecture 2400 thatincludes a managed network zone and an unmanaged network zone. As shown,the managed network zone includes a root node 2415, pool nodes 2420 and2425, and managed switching elements 2430-2455. These network elementsmay be implemented by different embodiments of corresponding networkelements that are described in this application. For example, the rootnode 2415 may be implemented by the root nodes described above byreference to FIG. 21 , the pool nodes 2420 and 2425 may be implementedby the pool nodes described above by reference to FIG. 16 , and themanaged switching elements 2430-2455 may be implemented by the switchingelement described above by reference to FIG. 12 .

The unmanaged network zone includes an extender 2410, switching elements1-n, and multiple end hosts. One of ordinary skill in the art willrealize that the unmanaged network zone may include any number ofdifferent networks and end hosts, as indicated by dashed lines in FIG.24 . In some embodiments, the extender 2410 in the unmanaged networkzone is configured before deploying the extender in the unmanagednetwork zone. For example, some embodiments require an IP address of anetwork controller (or a network controller of a control cluster) thatis will be controlling the extender 2410 to be specified (e.g., througha command line interface provided by the extender 2410).

Since the network elements (e.g., switching elements 1-n) in theunmanaged network zone are not used to implement logical switchingelements (i.e., not controlled by a network controller), the networkelements in the unmanaged network zone will not recognize logicalcontext tags defined for the managed network. Accordingly, someembodiments of the extenders 2405 and 2410 remove the logical contexttag from packets before sending the packets to the network elements ofthe unmanaged network zone. In some embodiments, the extender 2405removes the logical context tag from packets to be forwarded to theextender 2410 while, in other embodiments, the extender 2410 removes thelogical context tag from packets that the extender 2410 receives fromthe extender 2405 and that are to be forwarded to network elements inthe unmanaged network zone.

Conversely, some embodiments of the extenders 2405 and 2410 add logicalcontext tags to packets that are received from network elements in theunmanaged network zone and destined for the managed network zone. Forinstance, the extender 2410 of some embodiments may add a logicalcontext tag to a packet that the extender 2410 receives from one of thenetwork elements (e.g., switching elements 1-n). The logical context tagmay, in some embodiments, indicate that the packet belongs to a genericlogical context representing packets that originate from an unmanagednetwork that are destined for the managed network zone. In someembodiments, the extender 2410 adds the logical context tag to thepacket when the extender 2410 receives the packets from network elementsin the unmanaged network zone while, in other embodiments, the extender2405 adds the logical context tag to the packet when the extender 2405receives the packets from the extender 2410.

FIG. 25 conceptually illustrates a network architecture 2500 thatincludes a managed network zone and an unmanaged network zone, which arepart of a data center. In particular, FIG. 25 conceptually illustratesthe use of an extender to facilitate the implementation of a logicalswitching element that logically connects a tenant's machines that arespread across a managed network zone and an unmanaged network zone.

As illustrated in FIG. 25 , the managed network zone includes a rootnode 2505, a pool node 2510, managed switching elements 2515 and 2520,and machines 2525-2550. These network elements may be implemented bydifferent embodiments of corresponding network elements that aredescribed in this application. For instance, the root node 2505 may beimplemented by the root nodes described above by reference to FIG. 21 ,the pool node 2510 may be implemented by the pool nodes described aboveby reference to FIG. 16 , the managed switching elements 2515 and 2520may be implemented by the switching element described above by referenceto FIG. 12 , and the machines may be implemented by the machinesdescribe above by reference to FIG. 1 .

The unmanaged network zone includes an extender 2555, switching elements1-n, and multiple machines. One of ordinary skill in the will realizethat the unmanaged network zone may include any number of differentnetworks and end hosts, as indicated by dashed lines. In addition, FIG.25 illustrates that the managed network zone and the unmanaged networkare coupled to each other through network 2560. Specifically, the rootnode 2505 of the managed network zone and the extender 2555 of theunmanaged network zone are coupled to each other through the network2560. The network 2560 may be a layer 2 network (e.g., a local areanetwork (LAN)) in some embodiments while the network 2560 may be a layer3 network.

In some embodiments, the extender 2555 in the unmanaged network zone isconfigured before deploying the extender in the unmanaged network zone.For example, some embodiments require an IP address of a networkcontroller (or a network controller of a control cluster) that is willbe controlling the extender 2555 to be specified (e.g., through acommand line interface provided by the extender 2555).

Because the network elements (e.g., switching elements 1-n) in theunmanaged network zone are not used to implement logical switchingelements (i.e., not controlled by a network controller), the networkelements in the unmanaged network zone will not recognize logicalcontext tags defined for the managed network. Therefore, someembodiments of the extender 2555 removes the logical context tag frompackets before sending the packets to the network elements of theunmanaged network zone through the network 2560. In addition, theextender 2555 of some embodiments adds logical context tags to packetsthat are received from network elements in the unmanaged network zoneand destined for the managed network. For instance, the extender 2555 ofsome embodiments may add a logical context tag to a packet that theextender 2555 receives from one of the network elements (e.g., switchingelements 1-n). The logical context tag may, in some embodiments,indicate that the packet belongs to a generic logical contextrepresenting packets that originate from an unmanaged network. In someembodiments, the extender 2555 adds the logical context tag to thepacket when the extender 2555 receives the packets from network elementsin the unmanaged network zone that are destined for the managed networkzone.

Although FIG. 25 shows a managed network zone coupled to an unmanagednetwork through a root node in the managed network zone and an extenderin the unmanaged network zone, some embodiments may utilize an extenderin the managed network zone to couple the managed network zone to theunmanaged network, similar to the managed network zone illustrated inFIG. 24 . Furthermore, FIG. 25 illustrates the use of an extender tofacilitate the implementation of a logical switching element thatlogically connects one tenant's machines that are spread across amanaged network zone and an unmanaged network zone. However, theextender may utilized to facilitate the implementation of differentlogical switching elements that logically connects different tenant'smachines that are spread across a managed network zone and an unmanagednetwork zone.

FIG. 26 conceptually illustrates an example of mapping logical contexttags between managed networks and unmanaged networks. As mentionedabove, some embodiments of extenders add logical context tags to packetsand/or remove logical context tags from packets. FIG. 26 conceptuallyillustrates examples of such mappings. As shown, an extender 2630provides a communication bridge between a managed network zone and anunmanaged network zone. The managed network zone includes a set of rootnodes, a set of pool nodes, and a set of managed switching elements. Theunmanaged network zone includes a set of unmanaged switching elements.

In some embodiments, the extender 2630 receives packet from the managednetwork zone that includes a logical context tag. Referring to FIG. 26as an example, packet A includes a logical context tag, as indicated byan “ID” in the packet's header. When the extender 2630 receives thepacket A, the extender 2630 removes the logical context tag from thepacket A. As shown, when the extender 2630 sends the packet A to theunmanaged network zone, the packet A no longer has the “ID” logicalcontext tag.

The extender 2630 of some embodiments maps packets from the unmanagednetwork zone to the managed network zone. In some of these embodiments,the extender 2630 identifies a logical context for the packets and addsa logical context tag that represents the identified logical context.Referring to FIG. 26 as an example, when packet B is sent to theextender 2630, the packet B does not have a logical context tag. Whenthe extender 2630 receives the packet B, the extender 2630 identifies alogical context for the packet B (e.g., by matching the packet B againstflow entries) and adds a logical context tag that represents theidentified logical context of the packet B. As noted above, the logicalcontext tag may, in some embodiments, indicate that the packet B belongsto a generic logical context representing packets that originate from anunmanaged network. Then, the extender 2630 sends the packet B to themanaged network zone.

While FIG. 26 illustrates mapping of logical context tags betweenmanaged networks and unmanaged networks by an extender, some embodimentsimplement such functionality in a different network element. Forinstance, a root node to which the extender is connected may performlogical context tag mapping between managed networks and unmanagednetworks, in some embodiments.

FIG. 27 conceptually illustrates an architectural diagram of an extender2785 of some embodiments. As shown, the extender 2785 is similar to theVM 1285, which is described above by reference to FIG. 12 , except theextender 2785 is running on the extender 2785's own computing device(e.g., a x86 computing device) instead of a VM that is running on ahypervisor along with other VMs in a single host.

The extender 2785 essentially functions similar to the VM 1285, asexplained above. Thus, NICs 2710 and 2715 function similar to the NICs1210 and 1215, extender network stack 2740 functions similar to thehypervisor network stack 1240, PIF bridges 2755 and 2760 functionsimilar to the PIF bridges 1255 and 1260, integration bridge 2750functions similar to the integration bridge 1250, flow processor 2775functions similar to the flow processor 1275, and Openflow protocolmodule 2770 functions similar to the Openflow protocol module 1270.However, the extender 2785 of some embodiments serves different purposesin a managed network, as noted above, and, thus, may be configureddifferently by a network controller of the managed network.

FIG. 28 conceptually illustrates a network architecture 2800 fordistributing packet processing between pool nodes 2805 and 2810. Thisfigure shows the network architecture 2800 that includes the pool nodes2805 and 2810, software switching elements 2815-2825, and VMs 2830-2860.In this example, the software switching elements 2815-2825 are managedswitching elements and the VMs 2830-2860 run on the same host as thecorresponding software switching element. That is, VMs 2830-2840 arerunning on the same host as the software switching element 2815, the VM2845 is running on the same host as the software switching element 2820,and the VMs 2850-2860 are running on the same host as the softwareswitching element 2825.

As described above, a software switching element may be an OVS that runson a physical host in some embodiments. In this example, the softwareswitching elements 2815-2825 are OVSs that each runs a physical host. Onthe right side of FIG. 28 , a block diagram of the software switchingelement 2825 and the physical host on which the software switchingelement 2825 runs is shown. The physical host includes physical ports2865, hypervisor 2870, patch ports 2875, OVS 2880, patch ports 2895, andthe VMs 2850-2860. The physical ports 2865, hypervisor 2870, patch ports2875, OVS 2880, patch ports 2895, and the VMs 2850-2860 are similar tothe corresponding components illustrated in FIG. 11 .

To distribute packet processing between the pool nodes 2805 and 2810,each of the pool nodes 2805 and 2810 needs to be able to process a givenpacket. As such, the pool nodes 2805 and 2810 each include the same setof flow entries, in some embodiments. This way, either the pool node2805 or the pool node 2810 can process a given packet.

Moreover, each of the software switching elements 2815-2825 needs to beable to access both of the pool nodes 2805 and 2810 in some embodiments.As such, some embodiments couple the software switching elements2815-2825 to the pool nodes 2805 and 2810 using tunnels that areprovided by tunneling protocols that are described above by reference toFIGS. 6 and 7 . As shown in FIG. 28 , each of the software switchingelements 2815-2825 is coupled to each of the pool nodes 2805 and 2810through a tunnel. In addition, each of the software switching elements2815-2825 is also coupled to each of the other software switchingelements 2815-2825 through a tunnel (e.g., a layer 3 tunnel), and, thus,can each communicate with one another. These tunnels are indicated bydashed arrows. This way, each of the software switching elements2815-2825 is aware of the interface (e.g., VIF) through which each VM iscoupled, and, thus, has access to the MAC address associated with eachof the interfaces through which the VMs are coupled. The tunnelconfiguration between the pool nodes 2805 and 2810 and the softwareswitching elements 2815-2825 illustrated in FIG. 28 is referred to as afull tunnel mesh in some embodiments.

In some embodiments, software switching elements 2815-2825 send packetsto the pool nodes 2805 and 2810 through designated ports. The designatedports are referred to as uplink ports in some embodiments. As shown inFIG. 28 , the patch ports 2875 include uplink ports 2885 and 2890. Theuplink port 2885 corresponds to the pool node 2805 and the uplink port2890 corresponds to the pool node 2810. Therefore, when the softwareswitching element 2825 wants to send packet to the pool node 2805, thesoftware switching element 2825 sends the packet to the uplink port 2885and when the software switching element 2825 wants to send packet to thepool node 2810, the software switching element 2825 sends the packet tothe uplink port 2890. The hypervisor 2870 of some embodiments managesthe uplink ports 2885 and 2890 such that the uplink ports 2885 and 2890correspond to the correct physical ports 2865 for the packets to reachthe pool nodes 2805 and 2810.

As mentioned above, FIG. 28 illustrates a full tunnel mesh configurationbetween software switching elements and pool nodes in a managed network.However, different embodiments may use different tunnel configurationsbetween the software switching elements and the pool nodes. For example,some embodiments might implement a partial tunnel mesh configuration. Insome such embodiments, the pool nodes are divided into subsets of poolnodes and each subset of pool nodes handles a portion of the packetprocessing load.

As the number of pool nodes, root nodes, and/or managed switchingelements increases in a manage network utilizing a full tunnel meshconfiguration, the complexity of the configuration can increase and theresources for establishing tunnels can decrease. FIG. 29 conceptuallyillustrates a tunnel configuration for reducing the number of tunnelsbetween the pool nodes, root nodes, and/or managed switching elements inthe managed network while providing all the managed switching elementsaccess to the pool node and root nodes.

As illustrated in FIG. 29 , a managed network 2900 includes pool androot nodes 2910-2930 and cliques 2940 and 2950. For this example, a pooland root node is a physical host (e.g., a server computer) on which anOVS runs as a pool node and an OVS runs as a root node. In someembodiments, a clique includes two or more managed switching elementsthat are coupled to each other in a full tunnel mesh configuration.

Referring to FIG. 29 , the managed switching elements in the clique 2940are each coupled to each other through tunnels. Similarly, the managedswitching elements in the clique 2950 also are each coupled to eachother through tunnels. However, none of the managed switching elementsin the clique 2940 are coupled to any of the managed switching elementsin the clique 2950. Thus, a lower number of tunnels are utilized thanthe number of tunnels that would be required if the managed switchingelements in the cliques 2940 and 2950 were all configure in a fulltunnel mesh configuration. Furthermore, each managed switching elementin the cliques 2940 and 2950 are coupled to each of the pool and rootnodes 2910-2930 through a tunnel. Although only a single arrow is shownbetween the cliques 2940 and 2950 and each of the pool and root nodes2910-2930, these arrows actually represent the tunnels (three tunnels inthis example) from each of the managed switching elements in the cliques2940 and 2950 and the pool and root nodes 2910-2930.

FIG. 30 conceptually illustrates a process 3000 of some embodiments forprocessing packets. In some embodiments, the process 3000 is performedby each managed switching element in a managed network that employs thepool node distribution technique described above by reference to FIG. 28. That is, the pool nodes in the managed network each include the sameset of flow entries and each of the managed switching elements canaccess each of the pool nodes. In some embodiments, each of the managedswitching elements perform the process 3000 when performing the secondstage 1420 of the processing pipeline 1400, which is described above byreference to FIG. 14 .

The process 3000 is similar in many respects to the process 2000described above by reference to FIG. 20 . However, the process 3000includes an addition operation for determining a hash value to determinea pool node to which to send the packet.

The operations 3010-3050 of the process 3000 are the same as theoperations 2010-2050 of the process 2000. That is, the process 3000determines (at 3010) whether the packet has an unknown destination MACaddress. If the packet has an unknown destination MAC address, theprocess 3000 continues to 3060. Otherwise, the process 3000 determines(at 3020) whether the packet can be processed. If the packet cannot beprocessed, the process 3000 proceeds to 3060. If the process 3000determines that the packet can be processed, the process 3000 processes(at 3030) the packet and then the process 3000 determines (at 3040)whether the packet is a multicast or broadcast packet.

If the process 3000 determines that the packet is not a multicast orbroadcast packet, the process 3000 ends. Otherwise, the process 3000determines (at 3050) whether the packet needs further processing. If thepacket does not need further processing, the process 3000 ends.Otherwise, the process 3000 proceeds to 3060.

At 3060, the process 3000 applies a hash function on a set of fields ofthe packet. Different embodiments of the process 3000 apply a hashfunction on different sets of fields of the packet. For instance, someembodiments apply a hash function on the source MAC address of thepacket while other embodiments apply a hash function on the source IPaddress of the packet. In some embodiments, a hash function is appliedon the destination MAC address of the packet. Some embodiments may applya hash function on both the source MAC address and the source IPaddress. Other ways of applying a hash function on the packet arepossible in other embodiments.

Finally, the process 3000 forwards (at 3070) the packet to a pool nodebased on the hash of the packet. In some embodiments, the hash functionused to hash the packet may be defined based on the number of pool nodesfrom which to choose in the managed network. For instance, referring toFIG. 29 as an example, some embodiments may define a hash function thathashes to three different values that each correspond to each of thepool and root nodes 2910-2930. This way, a hash of a packet selects oneof the pool nodes based on the value of the hash of the packet. Afterthe process 3000 forwards the packet to the pool node, the process 3000ends.

FIG. 31 conceptually illustrates a block diagram of a switching element3100 of some embodiments that processes packets to determine a pool nodeto which to send the packet. As shown, the switching element 3100includes ingress ports 3110, egress ports 3120, a dispatch port 3130,forwarding tables 3140, a packet processor 3150, a hash function module3160, a range list module 3170, a virtualization application 3175, andpool nodes 3180-3190.

The ingress ports 3110, the egress ports 3120, the dispatch port 3130,and the forwarding tables 3140 are similar to the ingress ports 910, theegress ports 920, the dispatch port 930, and the forwarding tables 940,which are described above by reference to FIG. 9 . However, theforwarding tables 3140 include a set of flow entries for processingpackets to determine a pool node to which to send the packet.Specifically, the forwarding tables 3140 includes a flow entry thatspecifies a hash function to be performed on packet when the packet isidentified as a multicast packet, and flow entries that specify one ofthe pool node 3180-3190 to which to sent the packet based on a hashvalue.

In some embodiments, the packet processor 3150 is similar to the packetprocessor 1090, which is described above by reference to FIG. 10 . Thatis, the packet processor 3150 processes network data (e.g., packets)that the packet processor 3150 receives from the ingress ports 3110based on flow entries in the forwarding tables 3140. When the packetprocessor 3150 wants to apply a hash function to a packet, the packetprocessor 3150 sends a copy of the packet to the hash function module3160 and, in return, receives a hash value. In some cases, the packetprocessor 3150 sends the hash value to the range list module 3170, and,in return, receives a value that corresponds to a pool node in themanaged network.

In some embodiments, the hash function module 3160 performs a hashfunction on the packet and returns a hash value. As mentioned above,different embodiments define different types of hash functions that canbe applied on different sets of fields of the packet (e.g., the sourceMAC address, the source IP address, etc.). The hash function module 3160of some embodiments receives hash functions from the virtualizationapplication 3175.

The range list module 3170 of some embodiments restricts the hash valuesof the hash functions to a defined range of values. The range of valuescorresponds to the number of pool nodes in the managed network fromwhich a pool node can be selected. Some embodiments of the range listmodule 3170 restrict the hash values of the hash function to the definedrange of values by mapping hash values to a corresponding value in thedefined range of values.

In some embodiments, the virtualization application 3175 is similar tothe virtualization applications described above by reference to FIGS.2-5 . In addition, the virtualization application 3175 of someembodiments defines a range of values for the range list module 3170.When a pool node is added or removed from the managed network, thevirtualization application 3175 of some embodiments dynamicallyredefines the range of values to reflect the number of pool nodescurrently in managed network from which to select and provides theredefined range of values to the range list module 3170.

Further, the virtualization application 3175 sends defined hashfunctions to the hash function module 3160, in some embodiments. When apool node is added or removed (e.g., the pool node fails) from themanaged network, some embodiments of the virtualization application 3175alternatively, or in conjunction with redefining a range of values forthe range list module 3170, redefine a hash function and provide theredefined hash function to the hash function module 3160.

The following will describe an example packet processing operation todetermine a pool node to which to send a packet. When the switchingelement 3100 receives a packet through a port of the ingress ports 3110,the packet is forwarded to the packet processor 3150 to process. Thepacket processor 3150 matches the packet against the flow entries in theforwarding tables 3140 to process the packet. In this example, thepacket is a multicast packet and needs to be processed by a pool node inthe managed network. As such, the packet processor 3150 determines thatthe packet matches the first flow entry illustrated in the forwardingtables 3140. The first flow entry specifies to apply a hash function onthe packet in order to select a pool node from the pool nodes 3180-3190to which to sent the packet for processing.

The packet processor 3150 sends a copy of the packet to the hashfunction module 3160. The hash function module 3160 applies the definedhash function on the copy of the packet and returns a hash value to thepacket processor 3150. Then, the packet processor 3150 sends the hashvalue to the range list module 3170 to receive a value that correspondsto one of the pool nodes 3180-3190. When the range list module 3170receives the hash value from the packet processor 3150, the range listmodule 3170 identifies a value in a defined set of values to which thehash value maps and returns the identified value to the packet processor3150. For this example, the identified value is 2.

Next, the packet processor 3150 stores the value that the packetprocessor 3150 receives from the range list module 3170 in the packet(e.g., in a logical context tag or another field in the packet header).The packet processor 3150 then sends the packet to the dispatch port3130 for further processing. When the dispatch port 3130 receives thepacket, the packet is sent back to a port of the ingress ports 3110. Thepacket is then forwarded back to the packet processor 3150 forprocessing.

Alternatively, some embodiments of the packet processor 3150 store thevalue that the packet processor 3150 receives from the range list module3170 as metadata that is associated with (instead of stored in thepacket itself) and passed along with the packet. In some of theseembodiments, the packet processor 3150 sends the packet and theassociated metadata to the dispatch port 3130 for further processing.When the dispatch port 3130 receives the packet and the associatedmetadata, the packet and the associated metadata is sent back to a portof the ingress ports 3110. The packet and the associated metadata isthen forwarded back to the packet processor 3150 for processing.

The packet processor 3150 again matches the packet against the flowentries in the forwarding tables 3140 to process the packet. This time,the packet processor 3150 determines that the packet matches the thirdflow entry illustrated in the forwarding tables 3140. The third flowentry specifies that the packet be sent to uplink port 2, whichcorresponds to the pool node 3185 in this example. Accordingly, thepacket processor 3150 sends the packet to the port of the egress ports3120 that corresponds to the uplink port 2. In some embodiments, thepacket processor 3150 removes the value (“2” in this example) resultingfrom the hash operation from the packet's header before sending thepacket to the egress ports 3120.

IV. Defining Switching Infrastructures

The following section will describe several examples of operations thatare performed when a managed network is operating. Some of theoperations relate to pool node creation, root node creation, hashfunction updating, and network controller creation, among otheroperations.

FIG. 32 conceptually illustrates a process 3200 of some embodiments forcreating a managed network. In some embodiments, the process 3200 isperformed by a network controller, such as the ones described above byreference to FIGS. 2-5 , that is controlling a managed network. Thenetwork controller performs the process 3200 when the network controllerfirst starts up, in some embodiments. In some embodiments, thevirtualization application layer of the network controller performs theprocess 3200.

The process 3200 begins by determining (at 3210) whether the managednetwork needs switching elements. In some embodiments, switchingelements include pool nodes, root nodes, and extenders. The process 3200of some embodiments can determine whether the managed network needsswitching elements based on several factors. Examples of such factorsinclude the number of machines, VMs, hosts, and any other type ofnetwork host in the managed network, the number of managed switchingelements in the managed network, the attributes of the managed switchingelements (e.g., hardware switching element or software switchingelement, amount of memory, amount of processing power, etc.) in themanaged networks, the number of tenants in the managed network, etc.When the process 3200 determines that the managed network does not needswitching elements, the process 3200 proceeds to 3230.

When the process 3200 determines that the managed network needsswitching elements, the process 3200 creates (at 3220) a set ofswitching elements for the managed network. Some embodiments of theprocess 3200 determine the number of switching elements to create basedon the same or similar factors listed above for the operation 3210.

Next, the process 3200 creates (at 3230) tunnels in the managed network.As described in various sections above, different embodiments createtunnels for different purposes and in different situations. Forinstance, some embodiments use tunnels to connect pool nodes and managedswitching elements in a full tunnel mesh configuration in order todistribute packet processing between the pool nodes. Some embodimentsuse tunnels to form cliques of managed switching elements.

Finally, the process 3200 populates (at 3240) flow entries in themanaged switching elements and switching elements in the managednetwork. Flow entries specify operations for processing packets as thepackets flow through the various managed switching elements andswitching elements in the managed network. As such, the process 3200 ofsome embodiments determines and defines flow entries for each managedswitching element and switching element in the managed network. In someembodiments, flow entries are determined and defined based on the samefactors used in the operation 3210 described above. Some embodimentsalso take into account the switching elements, if any, that were createdat the operation 3220 and the tunnels that were created at the operation3230 in determining and defining the flow entries. After the process3200 determines and defines all the flow entries, the process 3200populates the flow entries into the respective managed switchingelements and switching elements (e.g., through a switching controlprotocol, such as the Openflow protocol). The process 3200 then ends.

At any given time while a managed network is operating, changes to themanaged network (e.g., machines added, machines removed, switchingelements added, switching elements removed, etc.) may occur. In someembodiments, the managed network may be reconfigured (e.g., by a networkcontroller managing the managed network) in response to a change. Forinstance, additions of machines to the managed network might requireadditional switching elements (e.g., managed switching elements, poolnodes, root nodes, etc.). Conversely, when machines are removed from themanaged network, switching elements might be removed from the managednetwork as well. Different embodiments consider any number of differentfactors in determine when and in what manner to respond to a change inthe managed switching element. Several of the following figuresillustrate examples of how a managed network may respond to changes thatoccur to the managed network.

FIG. 33 conceptually illustrates the creation of additional switchingelements in a managed network 3300 according to some embodiments of theinvention. In particular, FIG. 33 conceptually illustrates the creationof additional switching elements in the managed network 3300 at twostages 3310 and 3320 of the operation of the managed network 3300 inresponse to an increase in the number of machines in the managed network3300.

The first stage 3310 illustrates that the managed network 3300 includesa pool node 3330, managed switching elements 3340-3360, and machinesbelonging to a tenant A that are coupled to each of the managedswitching elements 3340-3360. In addition, the first stage 3310illustrates that tunnel is established between the each of the managedswitching elements 3340-3360 and the pool node 3330, and between themanaged switching element 3350 and the managed switching element 3360.

In the second stage 3320 of the managed network 3300, additionalmachines have been added to the managed network 3300. Specifically,machines that belong to a tenant B are now coupled to each of themanaged switching elements 3340-3360. In this example, the pool node3330 cannot handle processing load with the addition of tenant B'smachines. Therefore, a set of network controllers (not shown) that aremanaging the managed network 3300 determined that the managed network3300 requires another pool node 3380 to lessen the load on the pool node3330.

In this example, only one pool node can support each of the managedswitching elements 3340-3360. Therefore, the set of network controllersalso determined that the pool node 3380 will support the managedswitching element 3350. In response, the tunnel between the managedswitching element 3350 and the pool node 3330 is torn down and a tunnelbetween the managed switching element 3350 and the pool node 3380 isestablished. As a result, the pool node 3330 and the managed switchingelement 3340 will not be able to communicate with the pool node 3380 andthe managed switching elements 3350 and 3360. In addition, since thereare multiple tenants in the managed network 3300, logical contextlearning needs to be performed. Thus, the set of network controllersdetermined to create a root node 3370 to provide a communication bridgebetween the pool nodes 3330 and 3380 and to perform logical contextlearning. As shown, tunnels between the pool nodes 3330 and 3380 and theroot node 3370 are established.

FIG. 34 conceptually illustrates the addition of managed switchingelements and the creation of additional switching elements to a managednetwork 3400 according to some embodiments of the invention.Specifically, FIG. 34 conceptually illustrates the addition of managedswitching elements to and the creation of additional switching elementsin the managed network 3400 at two stages 3405 and 3410 of the operationof the managed network 3400 in response to an increase in the number ofmachines in the managed network 3400.

As shown in the first stage 3405, the managed network 3400 includes apool node 3420, cliques 3430 and 3440, and groups of machines 3450 and3460, which to a tenant A. Each of the cliques 3430 and 3440 includesthree managed switching elements that are coupled to each other withtunnels in a full tunnel mesh configuration. In addition, for each ofthe cliques 3430 and 3440, the managed switching elements each includethe same set of flow entries (not shown). As shown, the machines 3450are coupled to the clique 3430 and the machines 3460 are coupled to theclique 3440.

In this example, the pool node 3420 processes packets that the managedswitching elements in the cliques 3430 and 3440 cannot process. As such,the cliques 3430 and 3440 are each coupled to the pool node 3420 throughtunnels. That is, a tunnel is established between each of the managedswitching elements in the cliques 3430 and 3440 and the pool node 3420.

The second stage 3410 illustrates that additional groups of machines3480 and 3490 have been added to the managed network 3400. As shown, themachines 3480 are coupled to the managed switching elements in theclique 3430 and the machines 3490 are coupled to the managed switchingelements in the clique 3440. In some embodiments, the addition of themachines 3480 and 3490 increases the load on the three managed switchingelements in the cliques 3430 and 3440 that are illustrated in the firststage 3405. As a result, a set of network controllers (not shown) thatare managing the managed network 3400 determined that the managednetwork 3400 requires additional managed switching elements. Asillustrated in the second stage 3410 of FIG. 34 , the cliques 3430 and3440 now each include six managed switching elements in order to handlethe additional load of processing packets from the machines 3450, 3460,3480, and 3490. The six managed switching elements in the cliques 3430and 3440 are coupled to each other in a full tunnel mesh configuration(not shown) in some embodiments.

In some embodiments, the addition of the machines 3480 and 3490 and themanaged switching elements to the cliques 3430 and 3440 also increasesthe load on the pool node 3420. The pool nodes 3420 may not havesufficient resources (e.g., memory or data storage) to handle all thepackets that the managed switching elements in the cliques 3430 and 3440cannot handle. Thus, the set of network controllers has also determinedthat the managed network 3400 needs another pool node 3470. As shown inthe second stage 3410, the pool node 3470 has been created and added tothe managed network 3400. In this example, the packet processingdistribution technique described above by reference to FIG. 28 isutilized. Accordingly, as shown in FIG. 34 , the cliques 3430 and 3440are coupled to each of the pool nodes 3420 and 3470 (i.e., each of themanaged switching elements cliques 3430 and 3440 are coupled to each ofthe pool nodes 3420 and 3470). That way, the packet processing load isdistributed between the pool nodes 3420 and 3470.

FIGS. 33 and 34 illustrate example scenarios in which pool nodes and/orroot nodes are added to a managed network. In some embodiments, the poolnodes and/or root nodes are added to the managed network through manualdeployment. For example, the pool nodes and/or root nodes may require auser to power up and manually issue commands to specify the networkcontroller or control cluster that is managing the managed network inorder to add the pool nodes and/or root nodes to the managed network. Inother embodiments, the pool nodes and/or root nodes are automaticallydeployment and added (e.g., by the network controller or controlcluster) to the managed network.

As explained above, some embodiments use a hashing technique todistribute packet processing that managed switching elements cannothandle across several pool nodes in a managed network. FIG. 35conceptually illustrates an example of updating a hash function when apool node is added to a managed network. In particular, FIG. 35conceptually illustrates a switching element 3540 at three differentstages 3510-3530 of a hash function update operation. In someembodiments, the switching element 3540 is a software switching element(e.g., an OVS switch) while, in other embodiments, the switching element3540 is a hardware switching element. In other embodiments, theswitching element 3540 may be any other type of network element that canroute network data.

The first stage 3510 illustrates that the managed network includes theswitching element 3540 and pool nodes 3560 and 3570. As shown, theswitching element 3540 includes a forwarding plane 3550. The forwardingplane 3550 of some embodiments is similar to the forwarding plane 1170described above by reference to FIG. 11 . That is, in these embodiments,the forwarding plane 3550 processes network data that the switchingelement 3540 receives and determines where to route the network data.Since the packet processing is distributed between the pool nodes 3560and 3570, the pool nodes 3560 and 3570 include the same set of flowentries.

In addition, the forwarding plane 3550 includes a hash function X. Thehash function X represents is a hash function that the forwarding plane3550 uses to select one of the pool nodes 3560 and 3570 when theforwarding plane 3550 wants to send a packet to a pool node forprocessing. In this example, packet processing is distributed based onlogical datapaths. Therefore, different logical datapaths in a logicaldatapath set may be distributed to different pool nodes. The hashfunction X may be applied to data in the packet (e.g., a header field,such as a logical context tag) that represents the logical datapath towhich the packet belongs, in some embodiments. The first stage 3510shows that the hash function X is defined to map packets that belong tothe logical datapath of flow A to the pool node 3560, map packets thatbelong to the logical datapath of flow B to the pool node 3560, and mappackets that belong to the logical datapath of flow C to the pool node3570.

In the second stage 3520, another pool node 3580 is added to the managednetwork, as indicated by a box with dashed lines. The pool node 3580includes the same set of flow entries as the pool nodes 3560 and 3580.At this stage 3520, the hash function for selecting a pool node is stillhash function X. As shown, packets that belong to the logical datapathof flow A are still mapped to the pool node 3560, packets that belong tothe logical datapath of flow B are still mapped to the pool node 3560,and packets that belong to the logical datapath of flow C are stillmapped to the pool node 3570.

The third stage 3530 illustrates the switching element 3540 after thehash function X has been updated to a hash function Y in response to theaddition of the pool node 3580. In some embodiments, the hash function Yis provided to the switching element 3540 by a network controller thatmanages the switching element 3540. The hash function Y is defined toevenly distribute packets that belong to the logical datapaths A, B, andC. For this example, the hash function Y maps packets that belong to thelogical datapath of flow A to the pool node 3560, maps packets thatbelong to the logical datapath of flow B to the pool node 3570, and mapspackets that belong to the logical datapath of flow C to the pool node3580.

While FIG. 35 illustrates the update of a hash function for selecting apool node from a group of pool nodes, this method may be similarly usedin other embodiment as well. For instance, the hash function in the hashfunction module 3160 may also be updated (e.g., by the virtualizationapplication 3175) in a similar manner as described above.

FIG. 36 conceptually illustrates a process 3600 of some embodiments forupdating a hash function. In some embodiments, the process 3600 isperformed by a network controller that manages managed switchingelements in a managed network that employs a packet processingdistribution technique, such as the one described above by reference toFIG. 28 .

The process 3600 begins by determining (at 3610) whether a change in thestatus of pool nodes in the managed network has occurred. In someembodiments, a change in the status of the pool nodes includes a poolnode is added to the managed network, a pool node is removed from themanaged network, or a pool node in the managed network is notfunctioning. A change in the status of pool nodes in the managed networkmay include additional and/or other types of events in otherembodiments.

When the process 3600 determines that a change in the status of the poolnodes has occurred, the process 3600 updates (at 3620) the status ofuplink ports on the managed switching elements in the managed network.For instance, when a pool node is added to the managed network, theprocess 3600 of some embodiments updates the status of the uplink portson the managed switching elements to include another uplink port for thenewly added pool node. Conversely, when a pool node is removed from themanaged network, some embodiments of the process 3600 updates the statusof the uplink ports on the managed switching elements to remove anuplink port. Next, the process 3600 sends (at 3630) an updated hash flowentry to the managed switching elements. In some embodiments, the hashflow entry specifies the hash function for the managed switchingelements to select a pool node in the managed network to which to sendpackets that the managed switching elements cannot process. The process3600 then ends.

When the process 3600 determines that a change in the status of the poolnodes has not occurred, the process 3600 continues to 3640, the process3600 determines (at 3640) whether a hash error has occurred on one ofthe managed switching elements in the managed network. Examples of hasherrors include hash value collisions, hash values that are outside adefined range, etc. When the process 3600 determines that a hash errorhas occurred on one of the managed switching elements in the managednetwork, the process 3600 sends (at 3630) an updated hash flow entry tothe managed switching elements. As noted above, some embodiments sends ahash flow entry that specifies a hash function for the managed switchingelements to select a pool node in the managed network to which to sendpackets that the managed switching elements cannot process.Specifically, the process 3600 sends a hash flow entry that corrects thehash error. Then, the process 3600 ends.

In some embodiments, the process 3600 is constantly repeated while thenetwork controller is managing the managed switching elements in themanaged network in order to continue checking for changes in the statusof pool nodes in the managed network and updating the hash flow entriesin the managed switching elements accordingly. In other embodiments, theprocess 3600 is repeated at defined intervals (e.g., 1 minute, 5minutes, 30 minutes, 1 hour, etc.).

The above description of FIGS. 35 and 36 relate to updating hashfunctions when a pool node is added or removed to a managed network. Insome instances, a pool node is removed from a managed network becausethe pool node has failed. FIG. 37 conceptually illustrates an example ofpool node failure handling according to some embodiments of theinvention. As shown, a network architecture 3700 includes managedswitching elements 3705 and 3710, and pool nodes A-C. In this example,each of the arrows in FIG. 37 represents a tunnel.

Some embodiments utilize tunnel “bundling” as a pool node faulttolerance technique. In some such embodiments, each pool node in thenetwork is designated a failover pool node so that packets destined forthe failed pool node may quickly continue to be processed by the networkarchitecture. In some embodiments, the failover pool node is referred toas a secondary pool node and the pool node for which the failover poolnode is designated is referred to as a primary pool node.

Different embodiments designate secondary pool nodes for the primarypool nodes in the network differently. For instance, some embodimentsspecify, for a particular primary pool node, another primary pool nodein the network as a secondary pool node. FIG. 37A conceptuallyillustrates such an example. Specifically, FIG. 37A illustrates ahierarchy traversal table 3715 of the managed switching element 3705. Asshown, the primary pool node for the pool node 1 is the pool node A, theprimary pool node for the pool node 2 is the pool node B, and theprimary pool node for the pool node 3 is the pool node C. Additionally,the hierarchy traversal table 3715 specifies the secondary pool nodesfor each of the primary pool nodes 1-3. In particular, the secondarypool node for the pool node 1 is the pool node B, the primary pool nodefor the pool node 2 is the pool node C, and the primary pool node forthe pool node 3 is the pool node A. In this example, the managedswitching elements 3705 and 3710 monitor the pool nodes 1-3 in order todetect when one of the pool nodes 1-3 fails.

FIG. 37B conceptually illustrates the network architecture 3700 afterthe managed switching element 3705 has detected that a pool node hasfailed. In particular, the managed switching element 3705 has detectedthat the primary pool node for the pool node 2 (pool node B in thisexample) has failed. FIG. 37B also illustrates the hierarchy traversaltable 3715 of the managed switching element 3705 after the managedswitching element 3705 has modified the hierarchy traversal table 3715in response to the detected failure of the pool node 2. As shown, theprimary pool node for the pool node 2 is now pool node C, which waspreviously the secondary pool node for the pool node 2. Thus, when themanaged switching element 3705 determines that a packet is to be sent tothe pool node 2 for processing, the managed switching element 3705 sendsthe packet to the pool node C.

In addition, since the pool node B was designated as the secondary poolnode for the pool node 1, the managed switching element 3705 hasmodified the hierarchy traversal table 3715 to no longer specify asecondary pool node for the pool node 1. However, in some embodiments,the managed switching element 3705 automatically designates newsecondary pool nodes when a pool node fails. The managed switchingelement 3705, for example, may designate the pool node C as thesecondary pool node for the pool node 1 and designate the pool node A asthe secondary pool node for the pool node 2.

FIG. 37C conceptually illustrates the network architecture 3700 after anew pool node D has been inserted into the network architecture 3700.More specifically, the pool node D is specified as the primary pool nodefor the pool node 2, as illustrated by the hierarchy traversal table3715. FIG. 37C also illustrates that the managed switching element 3705has specified secondary pool nodes for the pool node 1 and the pool node2 upon detection of the addition of the pool node D. As shown in thehierarchy traversal table 3715, the pool node D is designated as thesecondary pool node for the pool node 1 and the pool node C isdesignated as the secondary pool node for the pool node 2.

Instead of specifying one of the primary pool nodes in the network as asecondary pool node of a particular primary pool node, some embodimentsmay provide backup pool nodes as secondary pool nodes. The backup poolnodes of some embodiments are configured to stand by and replace aprimary pool node when the primary pool node fails. FIG. 37Dconceptually illustrates an example of the network architecture 3700that employs backup pool nodes. As shown, FIG. 37D illustrates thehierarchy traversal table 3715. For this example, the hierarchytraversal table 3715 specifies the primary pool node for the pool node 1as the pool node A, the primary pool node for the pool node 2 as thepool node B, and the primary pool node for the pool node 3 as the poolnode C. In additional, the hierarchy traversal table 3715 specifies thesecondary pool node for pool node 1 as the pool node B, the primary poolnode for pool node 2 as the pool node C, and the primary pool node forpool node 3 as the pool node A.

FIG. 37E conceptually illustrates the network architecture 3700 afterthe managed switching element 3705 has detected that a pool node hasfailed. In this example, the managed switching element 3705 has detectedthat the primary pool node for the pool node 2 (pool node B in thisexample) has failed. FIG. 37E further shows the hierarchy traversaltable 3715 of the managed switching element 3705 after the managedswitching element 3705 has modified the hierarchy traversal table 3715in response to the detected failure of the pool node 2. As shown, theprimary pool node for the pool node 2 is now pool node N, which waspreviously the secondary pool node for the pool node 2. Thus, when themanaged switching element 3705 determines that a packet is to be sent tothe pool node 2 for processing, the managed switching element 3705 sendsthe packet to the pool node N.

FIG. 37F conceptually illustrates the network architecture 3700 after anew pool node P has been inserted into the network architecture 3700. Asshown, a pool node P has been inserted into the network architecture3700. More specifically, the pool node P is specified as the secondarypool node for the pool node 2, as illustrated by the hierarchy traversaltable 3715. In some embodiments, the managed switching element 3705 mayspecify the newly added pool node, the pool node P, as the primary poolnode for the pool node 2 and designate the pool node N back to the poolnode N's previously role as the secondary pool node for the pool node 2.

Moreover, by utilizing a tunnel bundling technique, the tunnels to thepool nodes and the pool nodes may be viewed as a single entity (a“bundle” of tunnels) from the perspective of the network controllers inthe network. Specifically, the network controllers view the managedswitching element as coupled to a single pool node through a singletunnel. In some such embodiments, the network controllers may send flowentries that only specify that packets be sent to a pool node instead ofhaving to determine the number of pool nodes in the network and tospecify pool node to which the packet be sent. In other words, themanaged switching elements are responsible for selecting a pool nodewhen a packet to be sent to a pool node for processing.

By having the managed switching elements 3705 and 3710 handle pool nodefailures, the network controller or control cluster managing the managednetwork does not need to specify new flow entries to the managedswitching elements 3705 and 3710 each time a pool node fails. Inaddition, the response time to a pool node failure is faster byimplementing this functionality in the managed switching elements 3705and 3710 instead of the network controller or control cluster.

FIG. 38 conceptually illustrates the creation of additional networkcontrollers to a control cluster for managing a managed network 3800according to some embodiments of the invention. Specifically, FIG. 38Aconceptually illustrates an example of creating additional networkcontrollers in the control cluster for the managed network 3800 at twostages 3810 and 3820 of the operation of the managed network 3800 inresponse to an increase in the number of machines in the managed network3800.

The first stage 3810 of FIG. 38A illustrates the managed network 3800.The managed network 3800 is similar to the managed network 3300illustrated in FIG. 33 except managed network 3800 also includes anetwork controller 3830. The network controller 3830 is similar to thenetwork controllers described above by reference to FIGS. 2-5 . At thisstage 3810, the network controller 3830 manages the pool node 3330 andthe managed switching elements 3340-3360.

The second stage 3820 of FIG. 38A is similar to the second stage 3320that is described above by reference to FIG. 33 , but the second stage3820 of the managed network 3800 shows additional machines added to themanaged network 3800 that belong to a tenant C. As shown machines thatbelong to tenant C are now coupled to each of the managed switchingelements 3350 and 3360.

Similar to the second stage 3320, the pool node 3330, at the secondstage 3820, cannot handle processing load with the addition of tenantB's and tenant C's machines. Therefore, the network controller 3830determined that the managed network 3800 requires another pool node 3380to lessen the load on the pool node 3330. As a result, the tunnelbetween the managed switching element 3350 and the pool node 3330 istorn down, a tunnel between the managed switching element 3350 and thepool node 3380 is established, and a root node 3380 is created toprovide a communication bridge between the pool nodes 3330 and 3380 andto perform logical context learning.

In addition, the second stage 3820 illustrates that another networkcontroller 3840 has been added to the control cluster. In someembodiments, the computation demands of a network controller 3830increases as the number of tenants increases in the managed network 3800since the network controller would have to implement a logical switchingelement for each additional tenant across the managed switching elementsin the managed network. Similarly, an increase in the number of machinesand/or switching elements in the managed network 3800 would increase thecomputational demands of the network controller 3830.

In this example, the network controller cannot handle the load ofmanaging managed network 3800 due to the addition of tenant B's andtenant C's machines to the managed network 3800. For instance, thenetwork controller 3830 would have to define logical datapath sets foreach of the tenants B and C in order to implement corresponding logicalswitching elements for the tenants across the managed switching elements3340-3360 in the managed network 3800. Therefore, the network controller3830 determined to add the network controller 3840 to assist in themanagement of the managed network 3800.

As shown, FIG. 38A illustrates a simple case of creating additionalnetwork controllers to a control cluster for managing a managed network.However, the addition of one network controller to the control clusterin this example may be problematic from a reliability point of view. Forexample, some embodiments employ a majority/minority technique formaintaining reliability of a control cluster. In some such embodiments,the network controllers communicate with each other and the controlcluster continues to operate as long as a majority (i.e., greater thanhalf) of the network controllers in the control cluster can communicatewith each other. Therefore, the control cluster can withstand a minority(i.e., less than half) of the network controllers in the control clusterfailing before the control cluster fails.

Referring to the example illustrated in FIG. 38A, the addition of onenetwork controller to the control cluster is thus problematic under themajority/minority technique. Specifically, while the addition of the onenetwork controller to the control cluster increases the compute capacityof the control cluster, the reliability of the control cluster isreduced because the number of points of failure in the control clusteris increased to two (i.e., a failure of any one of the two networkcontrollers causes the control cluster to fail) without an increase inthe number of failures that the control cluster can withstand (one inthis example).

Thus, in order to maximize reliability of the control cluster, additionsof network controllers to the control clusters are constrained tonumbers that maximizes the size of the minority of network controllersin the control cluster. FIG. 38B conceptually illustrates such anexample of creating additional network controllers in the controlcluster for the managed network 3800 at two stages 3850 and 3860 of theoperation of the managed network 3800 in response to an increase in thenumber of machines in the managed network 3800.

The first stage 3850 of FIG. 38B is similar to the first stage 3810illustrated in FIG. 38A. At this stage 3850, the network controller 3830manages the pool node 3330 and the managed switching elements 3340-3360.

The second stage 3860 of FIG. 38B is similar to the second stage 3820 ofFIG. 38A except the second stage 3860 of the managed network 3800 showstwo network controllers 3840 and 3870 added to the control cluster dueto the increased computation demands of the network controller 3830. Inthis example, utilizing majority/minority technique, the addition of thetwo network controllers 3840 and 3870 increases the compute capacity ofthe control cluster and increases the minority (from zero to one in thisexample) of the network controllers 3830, 3840, and 3870 in the controlcluster failing before the control cluster fails.

FIG. 38B shows one example of adding a number of network controllers toa control cluster in a manner that maximizes the reliability of thecontrol cluster, one of ordinary skill in the art will realize thatdifferent numbers of network controllers may be added to the controlcluster so that the reliability of the control cluster is maximized. Forexample, network controllers may be added to the control cluster so thatthe control cluster has an odd number of network controllers.

While some factors for determining whether to add a network controllerto a managed network have been described above, other embodiments mayconsider additional and/or other factors as well in such adetermination.

FIG. 38 illustrates an example scenario in which a network controller isadded to a managed network. In some embodiments, the network controlleris added to the managed network through manual deployment. For example,the network controller may require a user to power up and manually issuecommands to specify the network controller or control cluster that ismanaging the managed network in order to add the network controller tothe managed network. In other embodiments, the network controller isautomatically deployment and added (e.g., by the existing networkcontroller) to the managed network.

Some embodiments may provide a network controller fault tolerance methodfor handling the failure of a network controller. In some embodiments, alogical switching element is managed by only one network controller (buta network controller may manage more than one logical switchingelements). Thus, some of these embodiments specify, for a particularnetwork controller, another network controller as a failover networkcontroller in the event the particular network controller fails. In someembodiments, the failover network controller is referred to as asecondary network controller and the network controller for which thefailover network controller is specified is referred to as a primarynetwork controller.

FIG. 47 conceptually illustrates an example of network controllerfailure handling according to some embodiments of the invention. Asshown, a network architecture 4700 includes logical switching elements 1and 2, network controllers A-C, and managed network 4705. In addition,FIG. 47 illustrates a logical switching element master table 4710. Insome embodiments, each of the network controllers A-C stores the logicalswitching element master table 4710 and communicates with each other tosynchronize the contents of the logical switching element master table4710.

In FIG. 47A, the logical switching element master table 4710 specifiesthat the primary network controller for the logical switching element 1is the network controller A, the primary network controller for thelogical switching element 2 is the network controller B, and the primarynetwork controller for the logical switching element 3 is the networkcontroller C. In additional, the logical switching element master table4710 specifies that the secondary network controller for the logicalswitching element 1 is the network controller B, the secondary networkcontroller for the logical switching element 2 is the network controllerC, and the secondary network controller for the logical switchingelement 3 is the network controller A. For this example, the networkcontrollers A-C communicate with each other in order to detect when oneof the network controllers A-C fails.

FIG. 47B conceptually illustrates the network architecture 4700 afterthe network controllers B and C have detected that the networkcontroller A has failed. FIG. 47B also illustrates the logical switchingelement master table 4710 after the network controllers B and C havemodified the logical switching element master table 4710 in response tothe detected failure of the network controller A. As shown, the primarynetwork controller for the logical switching element 1 is now thenetwork controller B, which was previously the secondary networkcontroller for the logical switching element 1. As such, the networkcontroller B now manages the logical switching element 1.

Additionally, since the network controller A was designated as thesecondary network controller for the logical switching element 3, thenetwork controllers B and C have modified the logical switching elementmaster table 4710 to no longer specify a secondary network controllerfor the logical switching element 3. However, in some embodiments, thenetwork controllers B and C may automatically designate new secondarynetwork controllers when a network controller fails. For instance, thenetwork controllers B and C may specify the network controller C as thesecondary network controller for the logical switching element 1 andspecify the network controller B as the secondary network controller forthe logical switching element 3.

FIG. 47C conceptually illustrates the network architecture 4700 after anew network controller D has been added to the network architecture4700. In particular, the network controller D is specified as theprimary network controller for the logical switching element 1, asillustrated by the logical switching element master table 4710. FIG. 47Calso illustrates that the network controllers B and C have specifiedsecondary network controllers for the logical switching element 1 andthe logical switching element 3 upon detection of the addition of thenetwork controller D. As shown in the logical switching element mastertable 4710, the network controller B is designated as the secondarynetwork controller for the logical switching element 1 and the networkcontroller D is designated as the secondary network controller for thelogical switching element 3.

Although FIGS. 47A-C illustrate failure handling of a network controllerthat manages a logical switching element, some embodiments also providefailure handling of a network controller of a managed switching element.In some cases, a managed switching element of some embodiments ismanaged by only one network controller (but a network controller maymanage more than one managed switching elements). As such, someembodiments specify, for a particular network controller, anothernetwork controller as a secondary network controller in the event theparticular network controller fails.

FIG. 48 conceptually illustrates another example of network controllerfailure handling according to some embodiments of the invention. Asshown, a network architecture 4800 includes logical switching element4805, network controllers A-C, and managed switching elements 1-3. Inaddition, FIG. 48 illustrates a managed switching element master table4810. In some embodiments, each of the network controllers A-C storesthe managed switching element master table 4810 and communicates witheach other to synchronize the contents of the logical switching elementmaster table 4810.

In FIG. 48A, the managed switching element master table 4810 specifiesthat the primary network controller for the managed switching element 1is the network controller A, the primary network controller for themanaged switching element 2 is the network controller B, and the primarynetwork controller for the managed switching element 3 is the networkcontroller C. Additionally, the managed switching element master table4810 specifies that the secondary network controller for the managedswitching element 1 is the network controller B, the secondary networkcontroller for the managed switching element 2 is the network controllerC, and the secondary network controller for the managed switchingelement 3 is the network controller A. In this example, the networkcontrollers A-C communicate with each other in order to detect when oneof the network controllers A-C fails.

FIG. 48B conceptually illustrates the network architecture 4800 afterthe network controllers A and C have detected that the networkcontroller B has failed. Also, FIG. 48B illustrates the managedswitching element master table 4810 after the network controllers A andC have modified the managed switching element master table 4810 inresponse to the detected failure of the network controller B. As shown,the primary network controller for the managed switching element 2 isnow the network controller C, which was previously the secondary networkcontroller for the managed switching element 2. Accordingly, the networkcontroller C now manages the managed switching element 2.

Furthermore, since the network controller B was designated as thesecondary network controller for the managed switching element 1, thenetwork controllers A and C have modified the managed switching elementmaster table 4810 to no longer specify a secondary network controllerfor the managed switching element 1. However, the network controllers Aand C of some embodiments may automatically specify new secondarynetwork controllers when a network controller fails. For instance, thenetwork controllers A and C may specify the network controller C as thesecondary network controller for the managed switching element 1 andspecify the network controller A as the secondary network controller forthe logical switching element 2.

FIG. 48C conceptually illustrates the network architecture 4800 after anew network controller D has been added to the network architecture4800. In particular, the network controller D is specified as theprimary network controller for the managed switching element 2, asillustrated by the managed switching element master table 4810. FIG. 48Calso illustrates that the network controllers A and C have specifiedsecondary network controllers for the managed switching element 1 andthe managed switching element 2 upon detection of the addition of thenetwork controller D. As shown in the managed switching element mastertable 4810, the network controller D is designated as the secondarynetwork controller for the managed switching element 1 and the networkcontroller C is designated as the secondary network controller for themanaged switching element 2.

V. Logical Processing

FIG. 39 conceptually illustrates a process 3900 of some embodiments forprocessing a packet through a logical switching element that isimplemented across a set of managed switching elements in a managednetwork. In some embodiments, each managed switching element in themanaged network performs the process 3900 when the managed switchingelement receives a packet.

The process 3900 starts by mapping (at 3910) the packet to a logicalcontext. As noted above, a logical context of some embodimentsrepresents the state of the packet with respect to a logical switchingelement. The process 3900 maps the packet to the packet's logicalcontext in order to identify the stage in the logical switching elementthe packet is at.

Next, the process 3900 performs (at 3920) logical processing on thepacket. Different embodiments perform logical processing on the packetdifferently. For example, the logical switching element may beimplemented as a layer 2 switching element. In these cases, the logicalprocessing includes performing logical layer 2 operations on the packet,such as performing a logical layer 2 lookup on the packet to determinethe logical egress port of the logical switching element through whichto send the packet.

In some cases, the process 3900 performs only a portion of the logicalprocessing on the packet. For example, the process 3900 may startperforming the logical processing on the packet, but the process 3900does not complete the logical processing. Rather than waste the logicalprocessing that has already been performed on the packet, the process3900 modifies the logical context of the packet to indicate the stage inthe logical processing that the packet is at so that logical processingon the packet can resume where the logical processing left off the nexttime the logical processing is performed on the packet (e.g., by themanaged switching element that receives the packet next).

Other instances where the process 3900 performs only a portion of thelogical processing on the packet is when a portion of the logicalprocessing has already been performed on the packet (e.g., by a previousmanaged switching element). In these instances, the logical context ofthe packet, which was identified by the mapping of the packet to alogical context in the operation 3910, indicates the stage in thelogical processing that the packet is at. Accordingly, the process 3900resumes performing the logical processing on the packet at this point inthe logical processing.

After the process 3900 performs the logical processing (or a portion ofthe logical processing) on the packet, the process 3900 maps (at 3930)the result of the logical processing of the packet a correspondingphysical result. For example, when the result of the logical processingof the packet determines a logical port of the logical switching elementthrough which to send the packet, the process 3900 maps the logicalport(s) to a corresponding physical port(s) (e.g., a port of a managedswitching element that is used to implement the logical switchingelement) through which to send the packet. In some embodiments, thephysical port may be a physical port of a managed switching element thatis different from the managed switching element that is performing theprocess 3900.

Finally, the process 3900 performs (at 3940) physical processing on thepacket to determine the physical port of the managed switching elementthat is performing the process 3900 through which to send the packet sothe packet reaches the physical port(s) determined at the operation3930.

FIG. 40 conceptually illustrates a processing pipeline 4000 of someembodiments for processing a packet through a logical switching element.Specifically, the processing pipeline 4000 includes six stages 4020-4070for processing a packet through a logical switching element that isimplemented across a set of managed switching elements in a managednetwork. In some embodiments, each managed switching element in themanaged network that receives the packet performs the processingpipeline 4000 when the managed switching element receives the packet.

In some embodiments, a packet includes a header and a payload. Theheader includes, in some embodiments, a set of fields that containsinformation used for routing the packet through a network. Switchingelements may determine switching decisions based on the contained in theheader and may, in some cases, modify some or all of the header fields.As explained above, some embodiments determine switching decisions basedon flow entries in the switching elements' forwarding tables.

In some embodiments, the processing pipeline 4000 may be implemented byflow entries in the managed switching elements in the network. Forinstance, some or all of the flow entries are defined such that thepacket is processed against the flow entries based on the logicalcontext tag in the packet's header. Therefore, in some of theseembodiments, the managed switching elements are configured (e.g., by anetwork controller illustrated in FIGS. 1-5 ) with such flow entries.

As shown, FIG. 40 illustrates a set of ingress ports 4010, a set ofqueues 4080, and a set of egress ports 4090. The set of ingress ports4010 conceptually represent a set of ports (e.g., a tunnel port, NICs,VIFs, PIFs) of the managed switching element that is performing theprocessing pipeline 4000. The ingress ports 4010 are ports through whichthe managed switching element receives packets. The set of queues 4080conceptually represents a set of queues of the managed switching elementthat is performing the processing pipeline 4000. In some embodiments,the set of queues 4080 are for implementing resource control mechanisms,such as quality of service (QoS). The set of egress ports 4090conceptually represent a set of ports (e.g., a tunnel port, NICs, VIFs,PIFs) of the managed switching element that is performing the processingpipeline 4000. The egress ports 4090 are ports through which the managedswitching element sends packets. In some embodiments, at least one portin the set of ingress ports 4010 is also a port in the set of egressports 4090. In some embodiments, the set of ingress ports 4010 and theset of egress ports 4090 are the same set of ports. That is, the managedswitching element includes a set of ports that are used both to receivepackets and to send packets.

The first stage 4020 is similar to the first stage 1410 of theprocessing pipeline 1400, which is described above by reference to FIG.14 . At the stage 4020, ingress context mapping is performed on a packetto determine the logical context of the packet. In some embodiments, thefirst stage 4020 is performed when the logical switching elementreceives the packet (e.g., the packet is initially received by a managedswitching element in the network that implements the logical switchingelements). As noted above, a logical context, in some embodiments,represents the state of the packet with respect to the logical switchingelement. The logical context may, for example, specify the logicalswitching element to which the packet belongs, the logical port of thelogical switching element through which the packet was received, thelogical port of the logical switching element through which the packetis to be transmitted, the stage of the logical forwarding plane of thelogical switching element the packet is at, etc.

Some embodiments determine the logical context of a packet based on thesource MAC address of the packet (i.e., the machine from which thepacket was sent). Some embodiments perform the logical context lookupbased on the source MAC address of the packet and the inport (i.e.,ingress port) of the packet (i.e., the port of the managed switchingelement through which the packet was received). Other embodiments mayuse other fields in the packet's header (e.g., MPLS header, VLAN id,etc.) for determining the logical context of the packet.

After the first stage 4020 is performed, some embodiments store theinformation that represents the logical context in one or more fields ofthe packet's header. These fields may also be referred to as a logicalcontext tag or a logical context ID. Furthermore, the logical contexttag may coincide with one or more known header fields (e.g., the VLAN idfield) in some embodiments. As such, these embodiments do not utilizethe known header field or its accompanying features in the manner thatthe header field is defined to be used. Alternatively, some embodimentsstore the information that represents the logical context as metadatathat is associated with (instead of stored in the packet itself) andpassed along with the packet.

In some embodiments, the second stage 4030 is defined for the logicalswitching element. In some such embodiments, the second stage 4030operates on the packet's logical context to determine ingress accesscontrol of the packet with respect to the logical switching element. Forexample, an ingress ACL is applied to the packet to control the packet'saccess to the logical switching element when the logical switchingelement receives the packet. The ingress ACL may be defined to implementother ACL functionalities, such as counters, port security (e.g., allowpackets received through a port that originated only from a particularmachine(s)), and machine isolation (e.g., allow broadcast/multicastpackets received from a particular machine to be sent to only machinesthat belong to the same tenant or logical switching element), amongother ACL functionalities. Based on the ingress ACL defined for thelogical switching element, the packet may be further processed (e.g., bythe third stage 4040) or the packet may be dropped, for example.

In the third stage 4040 of the processing pipeline 4000, logicalprocessing is performed on the packet in the context of the logicalswitching element. In some embodiments, the third stage 4040 operates onthe packet's logical context to process and route the packet withrespect to the logical switching element. Different embodiments definelogical processing for the logical switching element differently. Forinstance, some embodiments define a logical layer 2 table for processingthe packet at layer 2 of the logical network. Alternatively, or inconjunction with the logical layer 2 table, some embodiments define alogical layer 3 table for processing the packet at layer 3 of thelogical network. Other embodiments may define other logical process forthe packet at the stage 4040.

The fourth stage 4050 of some embodiments is defined for the logicalswitching element. The fourth stage 4050 of some such embodimentsoperates on the packet's logical context to determine egress accesscontrol of the packet with respect to the logical switching element. Forinstance, an egress ACL may be applied to the packet to control thepacket's access out of the logical switching element after logicalprocessing has been performed on the packet. Based on the egress ACLdefined for the logical switching element, the packet may be furtherprocessed (e.g., sent out of a logical port of the logical switchingelement or sent to a dispatch port for further processing) or the packetmay be dropped, for example.

In the fifth stage 4060 of the processing pipeline 4000 is similar tothe third stage 1430 of the processing pipeline 1400, which is describedabove by reference to FIG. 14 . At the fifth stage 4050, egress contextmapping is performed to identify a physical result that corresponds tothe result of the logical processing of the packet. For example, thelogical processing of the packet may specify that the packet is to besent out of one or more logical ports (e.g., a logical egress port) ofthe logical switching element. As such, the egress context mappingoperation identifies a physical port(s) of one or more of the managedswitching elements that corresponds to the particular logical port ofthe logical switching element.

The sixth stage 4070 of the processing pipeline 4000 performs a physicalmapping based on the egress context mapping performed at the fifth stage4060. In some embodiments, the physical mapping determines operationsfor routing the packet to the physical port that was determined in thefifth stage 4060. For example, the physical mapping of some embodimentsdetermines one or more queues in the set of queues 4080 associated withone or more ports of the set of ports 4080 of the managed switchingelements that is performing the processing pipeline 4000 through whichto send the packet in order for the packet to reach the physical port(s)determined in the fifth stage 4060. This way, the managed switchingelements can route the packet along the correct path in the network forthe packet to reach the determined physical port(s). Also, someembodiments remove the logical context tag after the sixth stage 4070 iscompleted in order to return the packet to its original state before thepacket was processed by the processing pipeline 4000.

As mentioned above, in some embodiments, the processing pipeline 4000 isperformed by each managed switching element in the managed network thatis used to implement the logical switching element. The processingpipeline 4000 of some embodiments may be distributed across the managedswitching elements in the managed network. For example, in someembodiments, the second-fourth stages 4030-4050 are distributed acrossthe managed switching elements in the managed network. In some of theseembodiments, the managed switching element that initially receives thepacket may perform the first-sixth stages 4020-4070 and the remainingmanaged switching elements that subsequently receive the packet onlyperform the first, fifth, and sixth stages 4020, 4060, and 4070.

FIG. 41 conceptually illustrates a processing pipeline 4100 of someembodiments for processing a packet through a logical switching element.In particular, the processing pipeline 4100 includes four stages4120-4150 for processing a packet, by operating on a 64-bit logicalcontext tag of the packet, through a logical switching element that isimplemented across a set of managed switching elements in a managednetwork. In some embodiments, each managed switching element in themanaged network that receives the packet performs the processingpipeline 4100 when the managed switching element receives the packet.

As explained above, a packet, in some embodiments, includes a header anda payload. In some embodiments, the header includes a set of fields thatcontains information used for routing the packet through a network.Switching elements may determine switching decisions based on the fieldscontained in the header and may, in some cases, modify some or all ofthe header fields. As explained above, some embodiments determineswitching decisions based on flow entries in the switching elements'forwarding tables.

In this example, the 64-bit context tag is a field that is included inthe header of a packet. As shown, the 64-bit context tag includes a32-bit virtual routing function (VRF) field, a 16-bit logical inportfield, and a 16-bit logical outport field. The 32-bit VRF fieldrepresents the logical switching element to which the packet belongs andthe stage of the logical forwarding plane of the logical switchingelement the packet is at, the 16-bit logical inport field represents thelogical port of the logical switching element through which the packetwas received, and the 16-bit logical outport field represents thelogical port of the logical switching element through which the packetis to be transmitted.

In some embodiments, the processing pipeline 4100 may be implemented byflow entries in the managed switching elements in the network. Forinstance, some or all of the flow entries are defined such that thepacket is processed against the flow entries based on the 64-bit logicalcontext tag in the packet's header. Therefore, in some of theseembodiments, the managed switching elements are configured (e.g., by anetwork controller illustrated in FIGS. 1-5 ) with such flow entries.

As shown, FIG. 41 illustrates a set of ingress ports 4110, a set ofqueues 4180, and a set of egress ports 4190. The set of ingress ports4110, the set of queues 4180, and the set of egress ports 4190 aresimilar to the set of ingress ports 4010, the set of queues 4080, andthe set of egress ports 4090, respectively. The set of ingress ports4110 conceptually represent a set of ports (e.g., a tunnel port, NICs,VIFs, PIFs) of the managed switching element that is performing theprocessing pipeline 4100. The ingress ports 4110 are ports through whichthe managed switching element receives packets. The set of queues 4180conceptually represents a set of queues of the managed switching elementthat is performing the processing pipeline 4100. In some embodiments,the set of queues 4180 are for implementing resource control mechanisms,such as quality of service (QoS). The set of egress ports 4190conceptually represent a set of ports (e.g., a tunnel port, NICs, VIFs,PIFs) of the managed switching element that is performing the processingpipeline 4100. The egress ports 4190 are ports through which the managedswitching element sends packets. In some embodiments, at least one portin the set of ingress ports 4110 is also a port in the set of egressports 4190. In some embodiments, the set of ingress ports 4110 and theset of egress ports 4190 are the same set of ports. That is, the managedswitching element includes a set of ports that are used both to receivepackets and to send packets.

At the first stage 4120 of the processing pipeline 4100, a physical tological mapping is performed on a packet to determine the logicalcontext of the packet. In this example, the physical to logical mappingof the first stage 4120 determines the logical switching element towhich the packet belongs, the stage of the logical forwarding plane ofthe logical switching element the packet is at, and the logical port ofthe logical switching element through which the packet was received. Insome embodiments, the first stage 4120 is performed when the logicalswitching element receives the packet (e.g., the packet is initiallyreceived by a managed switching element in the network that implementsthe logical switching elements).

Different embodiments determine the logical context of a packet based ondifferent fields of the packet's header. For instance, as shown in FIG.41 , some embodiments determine the logical context of a packet based onthe source MAC address of the packet (i.e., the machine from which thepacket was sent), an inport (i.e., an ingress port in the set of ingressports 4110) of the packet (i.e., the physical port of the managedswitching element through which the packet was received), a VLAN id, the64-bit context tag, or any combination of the four fields.

After the first stage 4120 is performed, some embodiments store theinformation that represents the logical context in the packet's 64-bitlogical context tag, as illustrated by arrows from the stage 4120 to thecorresponding fields below. For example, the logical switching elementto which the packet belongs and the stage of the logical forwardingplane of the logical switching element the packet is at is stored in the32-bit VRF field, and the logical port of the logical switching elementthrough which the packet was received is stored in the 16-bit logicalinport field.

In some embodiments, the second stage 4130 is defined for the logicalswitching element. In this example, the second stage 4130 operates onthe packet's 64-bit logical context tag to determine access control ofthe packet with respect to the logical switching element. As shown byarrows pointing from the fields below to the stage 4130, an ACL operateson the 16-bit logical inport field and the 32-bit VRF field of thepacket's 64-bit logical context tag, which results in allowing thepacket to be further processed (e.g., by the third stage 4140), denyingthe packet (i.e., dropping the packet), or enqueuing the packet. In someembodiments, enqueuing the packet involves sending the packet to a queuein the set of queues 4180 that is associated with a port in the set ofegress ports 4190 for QoS purposes. In addition, the ACL may be definedto implement other ACL functionalities (not shown), such as counters,port security (e.g., allow packets received through a port thatoriginated only from a particular machine(s)), and machine isolation(e.g., allow broadcast/multicast packets received from a particularmachine to be sent to only machines that belong to the same tenant orlogical switching element), among ACL functionalities.

In the third stage 4140 of the processing pipeline 4100, the packet isprocessed against a logical L2 (layer 2) table to determine a logicaloutport, which corresponds to a logical port of the logical switchingelement through which the packet is to be sent. As shown by arrowspointing from the fields below to the stage 4140, the L2 table operateson the 16-bit logical inport field and the 32-bit VRF field of thepacket's 64-bit logical context tag in addition to the destination MACaddress of the packet. After the third stage 4140 is performed, someembodiments store the information that represents the determined logicaloutport in the 16-bit logical outport field of the packet's 64-bitlogical context tag, as illustrated by an arrow from the stage 4140 tothe outport field below.

At the fourth stage 4150 of the processing pipeline 4100, a logical tophysical mapping is performed to identify one or more physical ports ofone or more managed switching elements in the managed network thatcorresponds to the logical outport, which was determined in the thirdstage 4140, of the logical switching element. For this example, thefourth stage 4150 operates on the packet's 64-bit logical context tag toidentify one or more physical ports in the set of egress ports 4190through which to send the packet out in order for the packet to reachthe determined logical outport. As shown by arrows pointing from thefields below to the stage 4150, the fourth stage 4150 operates on the16-bit logical outport field and the 32-bit VRF field of the packet's64-bit logical context tag, which results in setting the 64-bit logicalcontext tag (e.g., saving the stage of the logical switching elementthat the packet is at, removing the 64-bit logical context tag), settingthe one or more queues in the set of queues 4180 associated with thephysical ports, and setting the one or more physical ports in the set ofegress ports 4190 through which to send the packet out.

As mentioned above, in some embodiments, the processing pipeline 4100 isperformed by each managed switching element in the managed network thatis used to implement the logical switching element. The processingpipeline 4100 of some embodiments may be distributed across the managedswitching elements in the managed network. For example, in someembodiments, the second and third stages 4130 and 4140 are distributedacross the managed switching elements in the managed network. In some ofthese embodiments, the managed switching element that initially receivesthe packet may perform the first-fourth stages 4120-4150 and theremaining managed switching elements that subsequently receive thepacket only perform the first and fourth stages 4120 and 4150.

In the above description of FIGS. 39, 40, and 41 , reference to“physical” components (e.g., physical switching element, physical ports,etc.) refers to the managed switching elements in the managed network.As explained above, a managed switching element may be a hardwareswitching element, a software switching element, or a virtual switchingelement. Thus, one of ordinary skill in the art will realize that thereference to a physical component is not meant to refer to an actualphysical component, but rather the reference is meant to distinguishfrom logical components (e.g., a logical switching element, a logicalport, etc.).

As mentioned above, some embodiments may distribute the processing of aprocessing pipeline across managed switching elements in a managednetwork. FIG. 42 conceptually illustrates distribution of logicalprocessing across managed switching elements in a managed networkaccording to some embodiments of the invention. In particular, FIG. 42conceptually illustrates a processing pipeline 4200 distributed acrosstwo managed switching elements 4210 and 4220. The processing pipeline4200 is similar to the processing pipeline 4000 described above byreference to FIG. 40 . Stage 4240 corresponds to the stage 4020, stage4250 corresponds to the stage 4030, stage 4260 corresponds to the stage4040, stage 4270 corresponds to the stage 4050, stage 4280 correspondsto the stage 4060, and stage 4290 corresponds to the stage 4070. Inaddition, FIG. 42 conceptually illustrates forwarding tables in themanaged switching elements 4210 and 4220 that are each implemented as asingle table and implementing multiple forwarding tables (e.g., using adispatch port, which is not shown) with the single table.

As illustrated in FIG. 42 , VM 1 is coupled to the managed switchingelement 4210, the managed switching element 4210 is coupled to themanaged switching element 4220, and the managed switching element 4220is coupled to VM 2. In this example, the VM 1 sends a packet 4230 to VM2 through a logical switching element that is implemented by the managedswitching elements 4210 and 4220.

As shown in the top half of FIG. 42 , the managed switching element 4210includes a forwarding table that includes rules (e.g., flow entries) forprocessing and routing the packet 4230. When the managed switchingelement 4210 receives the packet 4230 from the VM 1 through a VIF (notshown) of the managed switching element 4210, the managed switchingelement 4210 begins processing the packet 4230 based on the forwardingtables of the managed switching element 4210. The managed switchingelement 4210 identifies a record indicated by an encircled 1 (referredto as “record 1”) in the forwarding tables that implements the contextmapping of the stage 4240. The record 1 identifies the packet 4230'slogical context based on the inport, which is the VIF through which thepacket 4230 is received from the VM 1. In addition, the record 1specifies that the managed switching element 4210 store the logicalcontext of the packet 4230 in a set of fields (e.g., a VLAN id field) ofthe packet 4230's header. The record 1 also specifies the packet 4230 befurther processed by the forwarding tables (e.g., by sending the packet4230 to a dispatch port).

Based on the logical context and/or other fields stored in the packet4230's header, the managed switching element 4210 identifies a recordindicated by an encircled 2 (referred to as “record 2”) in theforwarding tables that implements the ingress ACL of the stage 4250. Inthis example, the record 2 allows the packet 4230 to be furtherprocessed and, thus, specifies the packet 4230 be further processed bythe forwarding tables (e.g., by sending the packet 4230 to a dispatchport). In addition, the record 2 specifies that the managed switchingelement 4210 store the logical context (i.e., the packet 4230 has beenprocessed by the second stage 4250 of the processing pipeline 4200) ofthe packet 4230 in the set of fields of the packet 4230's header.

Next, the managed switching element 4210 identifies, based on thelogical context and/or other fields stored in the packet 4230's header,a record indicated by an encircled 3 (referred to as “record 3”) in theforwarding tables that implements the logical L2 forwarding of the stage4260. The record 3 identifies the logical port of the logical switchingelement, which is implemented by the managed switching elements 4210 and4220, to which the packet 4230 is to be forwarded. The record 3 alsospecifies that the packet 4230 be further processed by the forwardingtables (e.g., by sending the packet 4230 to a dispatch port). Also, therecord 3 specifies that the managed switching element 4210 store thelogical context (i.e., the packet 4230 has been processed by the thirdstage 4260 of the processing pipeline 4200) in the set of fields of thepacket 4230's header.

Based on the logical context and/or other fields stored in the packet4230's header, the managed switching element 4210 identifies a recordindicated by an encircled 4 (referred to as “record 4”) in theforwarding tables that implements the egress ACL of the stage 4270. Inthis example, the record 4 allows the packet 4230 to be furtherprocessed and, thus, specifies the packet 4230 be further processed bythe forwarding tables (e.g., by sending the packet 4230 to a dispatchport). In addition, the record 4 specifies that the managed switchingelement 4210 store the logical context (i.e., the packet 4230 has beenprocessed by the fourth stage 4270 of the processing pipeline 4200) ofthe packet 4230 in the set of fields of the packet 4230's header.

In the fifth stage 4270 of the processing pipeline 4200, the managedswitching element 4210 identifies, based on the logical context and/orother fields stored in the packet 4230's header, a record indicated byan encircled 5 (referred to as “record 5”) in the forwarding tables thatimplements the context mapping of the stage 4280. In this example, therecord 5 identifies the VIF (not shown) of the managed switching element4220 to which the VM 2 is coupled as the port that corresponds to thelogical port of the logical switching element to which the packet 4230is to be forwarded. The record 5 additionally specifies that the packet4230 be further processed by the forwarding tables (e.g., by sending thepacket 4230 to a dispatch port).

Based on the logical context and/or other fields stored in the packet4230's header, the managed switching element 4210 then identifies arecord indicated by an encircled 6 (referred to as “record 6”) in theforwarding tables that implements the physical mapping of the stage4290. The record 6 specifies the port of the managed switching element4210 through which the packet 4230 is to be sent in order for the packet4230 to reach the VM 2. In this case, the managed switching element 4210is to send the packet 4230 out of the port (not shown) of managedswitching element 4210 that is coupled to the managed switching element4220.

As shown in the bottom half of FIG. 42 , the managed switching element4220 includes a forwarding table that includes rules (e.g., flowentries) for processing and routing the packet 4230. When the managedswitching element 4220 receives the packet 4230 from the managedswitching element 4210, the managed switching element 4220 beginsprocessing the packet 4230 based on the forwarding tables of the managedswitching element 4220. The managed switching element 4220 identifies arecord indicated by an encircled 1 (referred to as “record 1”) in theforwarding tables that implements the context mapping of the stage 4240.The record 1 identifies the packet 4230's logical context based on thelogical context that is stored in the packet 4230's header. The logicalcontext specifies that the packet 4230 has been processed by thesecond-fourth stages 4250-4270 of the processing pipeline 4200, whichwas performed by the managed switching element 4210. As such, the record1 specifies that the packet 4230 be further processed by the forwardingtables (e.g., by sending the packet 4230 to a dispatch port).

Next, the managed switching element 4220 identifies, based on thelogical context and/or other fields stored in the packet 4230's header,a record indicated by an encircled 2 (referred to as “record 2”) in theforwarding tables that implements the context mapping of the stage 4280.In this example, the record 2 identifies the VIF (not shown) of themanaged switching element 4220 to which the VM 2 is coupled as the portthat corresponds to the logical port of the logical switching element(which was determined by the managed switching element 4210) to whichthe packet 4230 is to be forwarded. The record 2 additionally specifiesthat the packet 4230 be further processed by the forwarding tables(e.g., by sending the packet 4230 to a dispatch port).

Based on the logical context and/or other fields stored in the packet4230's header, the managed switching element 4220 identifies a recordindicated by an encircled 3 (referred to as “record 3”) in theforwarding tables that implements the physical mapping of the stage4290. The record 3 specifies the port of the managed switching element4220 through which the packet 4230 is to be sent in order for the packet4230 to reach the VM 2. In this case, the managed switching element 4220is to send the packet 4230 out of the VIF (not shown) of managedswitching element 4220 that is coupled to the VM 2.

The above description of FIG. 42 illustrates a managed switching elementin a managed network that performs an entire logical processing of aprocessing pipeline of some embodiments. However, some embodiments maydistribute the logical processing of a processing pipeline acrossseveral managed switching element in a managed network. The followingfigure conceptually illustrates an example of such an embodiment. FIG.43 conceptually illustrates the distribution of logical processingacross managed switching elements in a managed network according to someembodiments of the invention. Specifically, FIG. 43 conceptuallyillustrates the processing pipeline 4200 distributed across the twomanaged switching elements 4210 and 4220.

FIG. 43 is similar to FIG. 42 except FIG. 43 conceptually illustratesthat the managed switching element 4210 performs only a portion of thelogical processing of the processing pipeline 4200 and the managedswitching element 4220 performs the remaining portion of the logicalprocessing of the processing pipeline 4200. As shown in the top half ofFIG. 43 , the managed switching element 4210 performs the contextmapping of the stage 4240, the ingress ACL of the stage 4250, thelogical L2 forwarding of the stage 4260, the context mapping of thestage 4280, and the physical mapping of the stage 4290. The managedswitching element 4210 does not perform the egress ACL of the stage4270, which is one of the stages of the logical processing of theprocessing pipeline 4200. Accordingly, when the managed switchingelement 4220 sends the packet 4230 to the managed switching element 4220(at the stage 4290), the logical context stored in the packet 4230'sheader specifies that the packet 4230 has been processed by the thirdstage 4260 of the processing pipeline 4200).

As illustrated in the bottom half of FIG. 43 , when the managedswitching element 4220 receives the packet 4230 from the managedswitching element 4210, the managed switching element 4220 beginsprocessing the packet 4230 based on the forwarding tables of the managedswitching element 4220. The managed switching element 4220 identifies arecord indicated by an encircled 1 (referred to as “record 1”) in theforwarding tables that implements the context mapping of the stage 4240.The record 1 identifies the packet 4230's logical context based on thelogical context that is stored in the packet 4230's header. The logicalcontext specifies that the packet 4230 has been processed by the secondand third stages 4250 and 4260 of the processing pipeline 4200, whichwas performed by the managed switching element 4210. As such, the record1 specifies that the packet 4230 be further processed by the forwardingtables (e.g., by sending the packet 4230 to a dispatch port).

Based on the logical context and/or other fields stored in the packet4230's header, the managed switching element 4220 identifies a recordindicated by an encircled 2 (referred to as “record 2”) in theforwarding tables that implements the egress ACL of the stage 4270. Inthis example, the record 2 allows the packet 4230 to be furtherprocessed and, thus, specifies the packet 4230 be further processed bythe forwarding tables (e.g., by sending the packet 4230 to a dispatchport). In addition, the record 2 specifies that the managed switchingelement 4220 store the logical context (i.e., the packet 4230 has beenprocessed by the fourth stage 4270 of the processing pipeline 4200) ofthe packet 4230 in the set of fields of the packet 4230's header.

Finally, the managed switching element 4210 performs the context mappingof the stage 4280 and the physical mapping of the stage 4290 is asimilar manner was that described above by reference to FIG. 42 .

While FIGS. 42 and 43 show examples of distributing logical processingacross managed switching elements in a managed network, in someinstance, some or all of the logical processing may need to be processedagain. For instance, in some embodiments, a root node does not preservethe logical context of a packet. Thus, when a pool node receives apacket from the root node of such embodiments (e.g., when a patch bridgeof a pool node receives a packet from a root bridge, which areillustrated in FIG. 22 ), the pool node may have to perform the logicalprocessing of the processing pipeline due to the lack of a logicalcontext in the packet.

FIG. 44 illustrates several example flow entries that implement aportion of a processing pipeline of some embodiments. In these exampleflow entries, a packet's logical context is stored in a VLAN id field ofthe packet's header. In addition, these examples use port 4000 as thedispatch port to which packets are sent for further processing. Some ofthe flow entries will be described by reference to FIG. 45 , whichconceptually illustrates a network architecture 4500 of someembodiments. Specifically, FIG. 45 conceptually illustrates a host 1that includes a managed switching element 1 to which VM 1 is coupledthrough a port 1 and a host 2 that includes a managed switching element2 to which VM 2 is couple through port (not shown) of the managedswitching element 2. The host 1 is coupled to the host 2 a tunnel. Asshown, the tunnel terminates at port 3 of the managed switching element1 of the host 1 and a port (not shown) of the managed switching element2. A pool node is coupled to the host 1 through a tunnel that terminatesat a port 2 of the managed switching element 1 and is coupled to thehost 2 through a tunnel that terminates at a port (not shown) of themanaged switching element 2. In this example, the flow entries arestored in the managed switching element 1, and, thus, are for processingpackets that are received by the managed switching element 1.

As shown, flow entry 1 is for performing physical to logical mapping(i.e., ingress context mapping). The flow entry 1 specifies that when apacket is received on port 1, the packet's VLAN id is to be modified to2057 and the packet is to be submitted to port 4000, which is thedispatch port. The VLAN id of 2057 represents the context of the packetand indicates that the packet has been received on port 1 of the managedswitching element 1.

Flow entry 2 is for modifying the packet's context to indicate that thepacket is at the start of logical processing (e.g., stages 4250-4270 ofthe processing pipeline 4200) of the processing pipeline. As shown, theflow entry 2 specifies that when a packet is received on port 4000 andthe packet's VLAN id is 2057, the packet's VLAN id is to be modified to2054 and the packet is to be submitted to port 4000, which is thedispatch port. The VLAN id of 2054 represents the context of the packetand indicates that the packet is at the start of the logical processingof the processing pipeline.

Next, flow entry 3 is for performing an ingress ACL lookup. As shown,the flow entry 3 specifies that when a packet is received on port 4000and the packet's VLAN id is 2054, the packet's VLAN id is to be modifiedto 2055 and the packet is to be submitted to port 4000, which is thedispatch port. The VLAN id of 2055 represents the context of the packetand indicates that the packet has been processed by the ingress ACL andallowed through the ingress ACL.

Flow entries 4-6 are for performing logical lookups. The flow entry 4specifies that when a packet is received on port 4000, the packet's VLANid is 2055, and the packet's destination MAC address is00:23:20:01:01:01, the packet's VLAN id is to be modified to 2056 andthe packet is to be submitted to port 4000, which is the dispatch port.The VLAN id of 2056 represents the context of the packet and indicatesthat the packet is to be sent to the VM 1.

The flow entry 5 specifies that when a packet is received on port 4000,the packet's VLAN id is 2055, and the packet's destination MAC addressis 00:23:20:03:01:01, the packet's VLAN id is to be modified to 2058 andthe packet is to be submitted to port 4000, which is the dispatch port.The VLAN id of 2058 represents the context of the packet and indicatesthat the packet is to be sent to the VM 2.

The flow entry 6 specifies that when a packet is received on port 4000,the packet's VLAN id is 2055, and the packet's destination MAC addressis ff:ff:ff:ff:ff:ff, the packet's VLAN id is to be modified to 2050 andthe packet is to be submitted to port 4000, which is the dispatch port.The VLAN id of 2050 represents the context of the packet and indicatesthat the packet is a broadcast packet.

As shown, flow entry 7 is for performing logical to physical mapping(i.e., egress context mapping). The flow entry 7 specifies that when apacket is received on port 4000, and the packet's VLAN id is 2056, thepacket's VLAN id is to be stripped (i.e., removed) and the packet is tobe submitted to port 1 which is the port to which VM 1 is coupled. Thus,the flow entry 7 is for sending the packet to VM 1.

Flow entry 8 is for performing logical to physical mapping (i.e., egresscontext mapping). As illustrated in FIG. 44 , the flow entry 8 specifiesthat when a packet is received on port 4000 and the packet's VLAN id is2058, the packet's VLAN id is to be modified to 2058 and the packet isto be submitted to port 3, which is the port to the tunnel (i.e., atunnel port) that couples the managed switching element 1 to the managedswitching element 2. As such, the flow entry 8 is for sending the packetto the host 2.

Next, flow entry 9 is for processing a broadcast packet. Specifically,the flow entry 9 specifies that when a packet is received on port 4000and the packet's VLAN id is 2050, the packet's VLAN id is to be modifiedto 2056 and the packet is to be submitted to port 4000, which is thedispatch port. In addition, the flow entry 9 specifies that when apacket is received on port 4000 and the packet's VLAN id is 2050, thepacket's VLAN id is to be modified to 2056 and a copy of the packet isto be submitted to port 4000. Therefore, the flow entry 9 is for sendinga broadcast packet to the VM 1 and to other VMs in the same logicalnetwork as the VM 1, which include the VM 2 in this example.

Flow entry 10 is for sending a broadcast packet to the pool node. Asshown in FIG. 44 , the flow entry 10 specifies that when a packet isreceived on port 4000 and the packet's VLAN id is 2051, the packet'sVLAN id is to be modified to 2050 and the packet is to be submitted toport 2, which is the port to the tunnel (i.e., a tunnel port) thatcouples the managed switching element 1 to the pool node. As mentionedabove, the VLAN id of 2050 represents the context of the packet andindicates that the packet is a broadcast packet.

As shown, flow entry 11 is for performing logical to physical mapping(i.e., egress context mapping). The flow entry 11 specifies that when apacket is received on port 3, which is the tunnel (i.e., a tunnel port)that couples the managed switching element 1 to the managed switchingelement 2, and the packet's VLAN id is 2056, the packet's VLAN id is tobe modified to 2056 and the packet is to be submitted to port 4000,which is the dispatch port. Therefore, the flow entry 11 is for sendingthe packet, which is received from the managed switching element 2, tothe VM 1.

Next, flow entry 12 is for performing logical to physical mapping (i.e.,egress context mapping). As illustrated, the flow entry 12 specifiesthat when a packet is received on port 2, which is the tunnel (i.e., atunnel port) that couples the managed switching element 1 to the poolnode, and the packet's VLAN id is 2056, the packet's VLAN id is to bemodified to 2056 and the packet is to be submitted to port 4000, whichis the dispatch port. As such, the flow entry 12 is for sending thepacket, which is received from the pool node, to the VM 1.

Flow entry 13 is for performing a logical lookup. Specifically, the flowentry 13 is for sending all packets with unknown destination MACaddresses to a pool node via an uplink. As shown in FIG. 44 , the flowentry 13 specifies that when a packet is received on port 4000 and thepacket's VLAN id is 2055, the packet's VLAN id is to be modified to 2049and the packet is to be submitted to port 4000, which is the dispatchport. The VLAN id of 2049 represents the context of the packet andindicates that the packet is a packet with an unknown MAC address. Inaddition, the flow entry 13 includes a priority value that is lower thatthe flow entries 4-6, which are also for performing logical lookups.Since the priority value of the flow entry 13 is lower than all theother flow entries, the flow entry 13 is evaluated after all the otherflow entries have been evaluated against the packet. Thus, the flowentry 13 is for sending a packet with an unknown MAC address to the poolnode.

Finally, flow entry 14 is for sending a packet with an unknown MACaddress to the pool node. As illustrated in FIG. 44 , the flow entry 14specifies that when a packet is received on port 4000 and the packet'sVLAN id is 2049, the packet's VLAN id is to be modified to 2049 and thepacket is to be submitted to port 2, which is the port to the tunnel(i.e., a tunnel port) that couples the managed switching element 1 tothe pool node. As mentioned above, the VLAN id of 2049 represents thecontext of the packet and indicates that the packet is a packet withunknown MAC address.

FIG. 44 illustrates that some embodiments may define a context tag foreach point in a processing pipeline for processing a packet through alogical switching element that is implemented across a set of managedswitching elements in a managed network. However, some such embodimentsmay not write the context of the packet to the packet after every pointin the processing pipeline. For instance, if several stages of theprocessing pipeline are defined to be performed by a particular managedswitching element (e.g., by the managed switching element that initiallyreceives the packet), some embodiments may skip the writing of thecontext tag until the last stage of the several stages of the processingpipeline has been performed. In this fashion, the managed switchingelement may function faster by not having to repeatedly read a contexttag and write a context tag at every point in the processing pipeline.

VI. Computer System

Many of the above-described features and applications are implemented assoftware processes that are specified as a set of instructions recordedon a computer readable storage medium (also referred to as computerreadable medium). When these instructions are executed by one or moreprocessing unit(s) (e.g., one or more processors, cores of processors,or other processing units), they cause the processing unit(s) to performthe actions indicated in the instructions. Examples of computer readablemedia include, but are not limited to, CD-ROMs, flash drives, RAM chips,hard drives, EPROMs, etc. The computer readable media does not includecarrier waves and electronic signals passing wirelessly or over wiredconnections.

In this specification, the term “software” is meant to include firmwareresiding in read-only memory or applications stored in magnetic storagewhich can be read into memory for processing by a processor. Also, insome embodiments, multiple software inventions can be implemented assub-parts of a larger program while remaining distinct softwareinventions. In some embodiments, multiple software inventions can alsobe implemented as separate programs. Finally, any combination ofseparate programs that together implement a software invention describedhere is within the scope of the invention. In some embodiments, thesoftware programs, when installed to operate on one or more electronicsystems, define one or more specific machine implementations thatexecute and perform the operations of the software programs.

FIG. 46 conceptually illustrates a computer system 4600 with which someembodiments of the invention are implemented. The electronic system 4600may be a computer, phone, PDA, or any other sort of electronic device.Such an electronic system includes various types of computer readablemedia and interfaces for various other types of computer readable media.Electronic system 4600 includes a bus 4605, processing unit(s) 4610, agraphics processing unit (GPU) 4620, a system memory 4625, a read-onlymemory 4630, a permanent storage device 4635, input devices 4640, andoutput devices 4645.

The bus 4605 collectively represents all system, peripheral, and chipsetbuses that communicatively connect the numerous internal devices of theelectronic system 4600. For instance, the bus 4605 communicativelyconnects the processing unit(s) 4610 with the read-only memory 4630, theGPU 4620, the system memory 4625, and the permanent storage device 4635.

From these various memory units, the processing unit(s) 4610 retrieveinstructions to execute and data to process in order to execute theprocesses of the invention. The processing unit(s) may be a singleprocessor or a multi-core processor in different embodiments. Someinstructions are passed to and executed by the GPU 4620. The GPU 4620can offload various computations or complement the image processingprovided by the processing unit(s) 4610.

The read-only-memory (ROM) 4630 stores static data and instructions thatare needed by the processing unit(s) 4610 and other modules of theelectronic system. The permanent storage device 4635, on the other hand,is a read-and-write memory device. This device is a non-volatile memoryunit that stores instructions and data even when the electronic system4600 is off. Some embodiments of the invention use a mass-storage device(such as a magnetic or optical disk and its corresponding disk drive) asthe permanent storage device 4635.

Other embodiments use a removable storage device (such as a floppy disk,flash drive, or ZIP® disk, and its corresponding disk drive) as thepermanent storage device. Like the permanent storage device 4635, thesystem memory 4625 is a read-and-write memory device. However, unlikestorage device 4635, the system memory is a volatile read-and-writememory, such a random access memory. The system memory stores some ofthe instructions and data that the processor needs at runtime. In someembodiments, the invention's processes are stored in the system memory4625, the permanent storage device 4635, and/or the read-only memory4630. For example, the various memory units include instructions forprocessing multimedia clips in accordance with some embodiments. Fromthese various memory units, the processing unit(s) 4610 retrieveinstructions to execute and data to process in order to execute theprocesses of some embodiments.

The bus 4605 also connects to the input and output devices 4640 and4645. The input devices enable the user to communicate information andselect commands to the electronic system. The input devices 4640 includealphanumeric keyboards and pointing devices (also called “cursor controldevices”). The output devices 4645 display images generated by theelectronic system. The output devices include printers and displaydevices, such as cathode ray tubes (CRT) or liquid crystal displays(LCD). Some embodiments include devices such as a touchscreen thatfunction as both input and output devices.

Finally, as shown in FIG. 46 , bus 4605 also couples electronic system4600 to a network 4665 through a network adapter (not shown). In thismanner, the computer can be a part of a network of computers (such as alocal area network (“LAN”), a wide area network (“WAN”), or an Intranet,or a network of networks, such as the Internet. Any or all components ofelectronic system 4600 may be used in conjunction with the invention.

Some embodiments include electronic components, such as microprocessors,storage and memory that store computer program instructions in amachine-readable or computer-readable medium (alternatively referred toas computer-readable storage media, machine-readable media, ormachine-readable storage media). Some examples of such computer-readablemedia include RAM, ROM, read-only compact discs (CD-ROM), recordablecompact discs (CD-R), rewritable compact discs (CD-RW), read-onlydigital versatile discs (e.g., DVD-ROM, dual-layer DVD-ROM), a varietyof recordable/rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.),flash memory (e.g., SD cards, mini-SD cards, micro-SD cards, etc.),magnetic and/or solid state hard drives, read-only and recordableBlu-Ray® discs, ultra density optical discs, any other optical ormagnetic media, and floppy disks. The computer-readable media may storea computer program that is executable by at least one processing unitand includes sets of instructions for performing various operations.Examples of computer programs or computer code include machine code,such as is produced by a compiler, and files including higher-level codethat are executed by a computer, an electronic component, or amicroprocessor using an interpreter.

While the above discussion primarily refers to microprocessor ormulti-core processors that execute software, some embodiments areperformed by one or more integrated circuits, such as applicationspecific integrated circuits (ASICs) or field programmable gate arrays(FPGAs). In some embodiments, such integrated circuits executeinstructions that are stored on the circuit itself.

As used in this specification and any claims of this application, theterms “computer”, “server”, “processor”, and “memory” all refer toelectronic or other technological devices. These terms exclude people orgroups of people. For the purposes of the specification, the termsdisplay or displaying means displaying on an electronic device. As usedin this specification and any claims of this application, the terms“computer readable medium” and “computer readable media” are entirelyrestricted to tangible, physical objects that store information in aform that is readable by a computer. These terms exclude any wirelesssignals, wired download signals, and any other ephemeral signals.

While the invention has been described with reference to numerousspecific details, one of ordinary skill in the art will recognize thatthe invention can be embodied in other specific forms without departingfrom the spirit of the invention. In addition, a number of the figures(including FIGS. 15, 20, 30, 32, 36, and 39 ) conceptually illustrateprocesses. The specific operations of these processes may not beperformed in the exact order shown and described. The specificoperations may not be performed in one continuous series of operations,and different specific operations may be performed in differentembodiments. Furthermore, the process could be implemented using severalsub-processes, or as part of a larger macro process.

We claim:
 1. For a managed non-edge switching element (MNESE), a methodfor processing packets for a logical datapath set (LDPS) implemented bythe MNESE and a plurality of managed edge switching elements (MESEs),the method comprising: at a first MESE coupled to a source machine:providing a first packet, having a destination address associated withthe LDPS, along a first tunnel to the MNESE after a determination thatthe first MESE does not know the destination address of the first packetmade by performing a lookup operation on a forwarding table to determinewhether the destination address is known; and providing other packetswith destination addresses associated with the LDPS along other tunnelsto other MESEs when the first MESE knows the destination addresses ofthe other packets; and at the MNESE: receiving the first packet from thefirst MESE, wherein each particular MESE that implements the LDPS has atleast one machine associated with the LDPS coupled directly to theparticular MESE; and forwarding the first packet to the destinationaddress through a second MESE of the plurality of other MESEs, whereinthe MNESE stores data for forwarding packets to a plurality ofdestination addresses and does not receive packets directly from themachines associated with the LDPS.
 2. The method of claim 1, wherein theMNESE does not receive a second packet for which the first MESEdetermines that the destination is known by performing a lookupoperation on a forwarding table that includes an entry that matches amedia access control (MAC) address that is specified in a header of thesecond packet.
 3. The method of claim 2, wherein the MAC address is adestination MAC address.
 4. The method of claim 1, wherein thedestination address of the first packet is determined to be unknown whenthe forwarding table does not include an entry that matches a MACaddress that is specified in a header of the first packet.
 5. The methodof claim 1, wherein the plurality of MESEs comprises a set of managedsoftware switching elements.
 6. The method of claim 1, wherein theplurality of MESEs comprises a set of managed virtual switchingelements.
 7. The method of claim 1, wherein the plurality of MESEscomprises a set of managed hardware switching elements.
 8. The method ofclaim 1 further comprising: when the MNESE does not know the destinationaddress, flooding a network with a set of packets, each particularpacket in the set of packets including a broadcast MAC address specifiedin the header of the particular packet as the particular packet'sdestination MAC address; and using a reply to the set of packets toidentify the second MESE as an MESE in the plurality of MESEs associatedwith the destination address of the first packet, wherein the forwardingcomprises forwarding the first packet to the destination address throughthe second MESE after identifying the second MESE as being associatedwith the destination address.
 9. The method of claim 1 furthercomprising identifying the destination address of the packet beforetransmitting the first packet to the second MESE.
 10. A system forprocessing packets for a logical datapath set (LDPS) implemented by amanaged non-edge switching element (MNESE) and a plurality of managededge switching elements (MESEs), the system comprising: a first MESE ofthe plurality of MESEs that is coupled to a source machine and that isconfigured (i) to provide a first packet, having a destination addressdetermined not to be known to the first MESE based on the first MESEperforming a lookup operation on a forwarding table to determine whetherthe destination address is known, along a first tunnel to the MNESE and(ii) to provide other packets associated with the LDPS and havingdestination addresses determined to be known to the first MESE alongother tunnels to other MESEs of the plurality of MESEs, wherein eachparticular MESE that implements the LDPS has at least one machineassociated with the LDPS coupled directly to the particular MESE,wherein the first MESE is stored on a non-transitory machine-readablemedium of a computing device and executed by at least one processingunit of the computing device; and the MNESE that is configured (i) toreceive the first packet from the first MESE along the first tunnel, and(ii) to forward the first packet to the destination address through asecond MESE of the plurality of other MESEs, wherein the MNESE storesdata for forwarding packets to a plurality of destination addresses anddocs not receive packets directly from the machines associated with theLDPS.
 11. The system of claim 10, wherein the MNESE does not receive asecond packet for which the first MESE determines that the destinationis known by performing a lookup operation on a forwarding table thatincludes an entry that matches a media access control (MAC) address thatis specified in a header of the second packet.
 12. The system of claim11, wherein the MAC address is a destination MAC address.
 13. The systemof claim 10, wherein the destination address of the first packet isdetermined to be unknown when the forwarding table does not include anentry that matches a MAC address that is specified in a header of thefirst packet.
 14. The system of claim 10, wherein the plurality of MESEscomprises a set of managed software switching elements.
 15. The systemof claim 10, wherein the plurality of MESEs comprises a set of managedvirtual switching elements.
 16. The system of claim 10, wherein theplurality of MESEs comprises a set of managed hardware switchingelements.
 17. The system of claim 10 wherein: when the MNESE does notknow the destination address, the MNESE floods a network with a set ofpackets, each particular packet in the set of packets including abroadcast MAC address specified in the header of the particular packetas the particular packet's destination MAC address; and the MNESE uses areply to the set of packets to identify the second MESE as an MESE inthe plurality of MESEs associated with the destination address of thefirst packet, wherein the forwarding comprises forwarding the firstpacket to the destination address through the second MESE afteridentifying the second MESE as being associated with the destinationaddress.
 18. The system of claim 10, wherein the MNESE identifies thedestination address of the first packet before transmitting the firstpacket to the second MESE.